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LOCATING  AND  SEARCHING  ELECTRONIC  DOCUMENTS: 
A  USER  STUDY  OF  SUPPLY  PUBLICATIONS 
IN  THE  UNITED  STATES  MARINE  CORPS 


ABSTRACT 


This  exploratory  study  assesses  the  usability  of  an  online  policy  document  system, 
specifically  with  company  grade  supply  officers  in  the  United  States  Marine  Corps. 
Using  common  work  scenarios,  users  were  asked  to  search  for  specific  policy  documents. 
Their  efforts  to  navigate  through  the  system  were  recorded  and  analyzed  to  better 
understand  areas  in  the  system  that  posed  challenges  to  supply  officers’  efforts  at  policy 
retrieval  and  understanding.  The  primary  purpose  of  this  study  is  to  provide 
recommendations,  based  on  the  test  findings,  that  will  address  opportunities  to  enhance 
the  effectiveness  and  efficiency  of  the  current  electronic  publication  system.  Based  on 
this  analysis,  the  second  purpose  of  this  study  was  to  provide  recommendations  to 
enhance  the  effectiveness,  efficiency,  and  user  satisfaction  with  the  electronic  document 
system. 

Five  participants  were  observed  as  they  used  electronic  policy  and  regulatory 
documents  related  to  financial  management  and  property  control  functions.  The 
participants  were  chosen  because  they  represent  the  user  population  and  are  familiar  with 
the  electronic  document  system.  Each  participant  conducted  typical  search  and  retrieval 
tasks  using  think-aloud  protocols.  Each  session  was  videotaped,  and  participants  were 
interviewed  afterwards. 

In-depth  analysis  of  the  data  indicates  that  participants  are  generally  satisfied  with 
the  system,  but  significant  opportunities  exist  to  improve  its  effectiveness  and  efficiency. 
Most  notable  recommendations  are:  increase  search  capability,  provide  consolidated 
offering  of  content,  allow  personalization  and  portability,  improve  ability  to  see 
context/location  within  documents,  create  additional  linking  mechanisms,  and  migrate 
away  from  .pdf  file  fonnat.  Additional  recommendations  address  the  need  for  iterative 
testing  and  triangulation  of  problem  areas  to  increase  data  reliability. 
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I.  INTRODUCTION 


A.  PURPOSE 

A  key  function  of  a  supply  officer’s  duties  is  to  retrieve,  interpret,  and  advise  on 
new  and  existing  military  policy.  Traditionally,  such  policies  were  housed  in  a  paper- 
based  library;  however,  in  the  past  10  years  the  policy  libraries  have  been  migrating  to  an 
electronic  system,  accessible  via  the  internet.  The  purpose  of  the  current  study  is  to  assess 
the  usability  of  an  online  policy  document  system,  specifically  with  company  grade 
supply  officers  in  the  United  States  Marine  Corps.  Using  common  work  scenarios,  users 
were  asked  to  search  for  specific  policy  documents.  Their  efforts  to  navigate  through  the 
system  were  recorded  and  analyzed  to  better  understand  areas  in  the  system  that  posed 
challenges  to  supply  officers’  efforts  at  policy  retrieval  and  understanding.  Based  on  this 
analysis,  the  second  purpose  of  this  study  was  to  provide  recommendations  to  enhance 
the  effectiveness,  efficiency,  and  user  satisfaction  with  the  electronic  document  system. 

B.  BACKGROUND 

In  this  section,  I  discuss  the  need  to  study  the  electronic  supply  policy  document 
system  to  demonstrate  the  effectiveness  and  efficiency  of  the  existing  tool.  First,  I 
provide  an  historical  context  by  addressing  the  significance  of  supply  to  Marine  Corps 
commands.  Second,  I  discuss  the  importance  of  locating  and  retrieving  policy  documents 
for  supply  officers.  Third,  I  provide  an  overview  of  the  migration  from  a  paper-based  to 
an  electronic-based  policy  document  system.  Fourth,  I  present  the  business  case  analysis 
that  discusses  the  importance  of  usability  for  aiding  supply  officers  in  their  use  of 
electronic  policy  documents.  Fifth,  I  provide  an  overview  of  usability  research.  Sixth,  I 
present  the  research  questions  that  governed  this  study.  Finally  I  provide  an  overview  of 
the  project. 

Throughout  this  research,  I  shall  limit  discussion  to  Marine  Corps  supply  officers 
and  the  publications  and  training  related  to  their  duties.  Although  supply  functionality  is 
largely  executed  by  capable  and  well-trained  enlisted  Marines,  the  supply  officer  has  a 
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direct  and  potentially  far-reaching  impact  on  command  operations  through  an  advisory 
role  to  the  commander  and  supervisory  role  of  supply  operations.  Enlisted  Marines 
contribute  enormously  toward  that  goal,  yet  the  supply  officer  is  ultimately  responsible. 
For  this  reason,  supply  officers  and  the  policy  documents  they  use  to  conduct  their  duties 
were  selected  for  this  study.1  The  participants  of  the  study  are  discussed  further  in 
Chapter  II. 

1.  Significance  of  Supply  &  Supply  Policy 

Supply  is  a  crucial  element  in  the  operational  readiness  of  any  command.  In  fact, 
most  readiness  reporting  includes  supply  readiness  and  maintenance  readiness  as  primary 
measures  of  overall  command  readiness.  The  significance  of  supply  is  discussed  in 
Marine  Corps  Order  4400.163,  which  states  that  “Massive  manpower  is  not  the  prime 
asset  of  the  United  States;  this  nation’s  prime  asset  is  skilled  manpower  equipped  with 
the  proper  tools.”2 3  Equipping  is  a  complex  function  that  encompasses  over  $336.1  trillion 
dollars  in  Department  of  Defense  (DoD)  assets.  Consequently,  it  is  critical  that  supply 

3 

management  be  effective,  both  for  national  security  and  economic  reasons. 

Ground  supply  officers  supervise  the  execution  of  policies  and  procedures  that 
govern  the  complex  function  of  Supply.  To  execute  these  policies,  officers  rely  on  the 
expertise  and  advice  of  enlisted  personnel.  Supply  officers  also  consult  with  peers, 
higher  authority,  and  others  to  help  guide  their  duties.  Although  enlisted  personnel  and 
other  officers  assist  the  supply  officers  with  policy  retrieval  and  interpretation,  proficient 
supply  officers  must  be  capable  of  retrieving  and  understanding  policy  documents 
themselves  to  conduct  their  functions  as  an  advisor,  trainer,  and  supervisor.  This 
capability  is  necessary  so  that  supply  officers  can  advise  and  make  recommendations  to 
commanders  regarding  policy,  and  take  corrective  or  preventive  action  when  they 


1  As  a  Marine  Corps  Supply  Officer  for  seven  years,  I  have  first-hand  knowledge  of  the  domain, 
responsibilities,  and  importance  of  the  Supply  Officer  contribution. 

^  Marine  Corps  Order  4400.163,  Department  of  Defense  Supply  Management  Reference  Book,  Section 
1-2  (1985).  http://www. usmc. mil/directiv. nsf/l?Open  View&Count=3 000  (accessed  November  15,  2007). 

3  Ibid.,  section  1-3. 
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encounter  detrimental  supply  situations.4  An  effective  policy  document  system,  then,  is 
one  that  will  help  Supply  officers  achieve  their  goal  of  retrieving,  understanding,  and 
advising  others  on  matters  of  supply  policy. 

2.  Locating  and  Retrieving  Policy  Information 

Under  the  expansive  policy  document  system,  the  procedural  answers  supply 
officers  seek  can  be  obscured  by  the  complexity  of  the  system  itself.  At  a  minimum,  a 
user  who  wishes  to  retrieve  and  explore  policy  and  regulations  related  to  a  specific  task 
must: 

1 .  Determine  relevant  publications  governing  the  task, 

2.  Possess  or  locate  the  applicable  publications, 

3.  Search  for  infonnation  related  to  the  task  from  within  those  publications, 

4.  Comprehensively  integrate  the  results,  and 

5.  Interpret  the  intent  of  the  author 


In  addition  to  an  already  complex  process,  the  characteristics  of  the  electronic 
document  system  further  complicate  the  task.  For  example,  the  officer’s  effort  may  be 
complicated  because:  a)  documents  originate  from  multiple  sources  and  various  levels  of 
authority;  b)  policy  information  is  parsed,  segregated,  or  falls  in  overlapping  topic  areas; 
c)  policies  are  in  constant  flux  due  to  frequent  updates,  deletions,  clarifications,  or  other 
additional  alterations;  and  d)  policies  are  extensively  cross-referenced  both  within 
documents  and  amongst  documents.  Given  this  complexity,  it  is  important  that  the 
electronic  document  system  be  as  easy  to  use  as  possible. 


4  Marine  Corps  Order  P4400.150E  w/Erratum  and  change  1-2,  Marine  Corps  Consumer  Level  Supply 
Policy  Manual,  Section  1003.2a  (2001)  (hereafter  cited  as  MCO  P4400.150E). 
http://www.usmc. mi  l/directiv.nsf/l?OpenView&Count=3QOO  (accessed  November  15,  2007). 
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3.  Paper-Based  Policy  Document  System 


Historically,  Marine  Corps  supply  officers  were  required  to  have  physical  copies 
of  all  relevant  publications.5  This  collection  of  policy  documents  (known  as  a 
publication  library)  alleviated  the  burden  of  seeking  out  policy  references  for  recurring 
tasks  by  providing  ready  reference  to  the  policy  and  regulatory  guidance  right  off-the- 
shelf. 

In  some  ways,  the  paper-based  system  allowed  users  to  control  aspects  of 
effectiveness  and  efficiency.  By  physically  manipulating  their  personal  copies,  users 
were  able  to  organize  them  in  a  manner  they  deemed  useful  and  efficient.  Users  could 
also  bookmark,  highlight,  and  tab  relevant  or  repetitive  areas  of  interest  for  future  use. 
Furthermore,  the  paper-based  system  provided  users  the  ability  to  search  for  information 
in  familiar  ways  that  are  traditionally  employed  with  paper  documents  and  books  (e.g., 
indexes,  table  of  contents,  scanning).  A  major  drawback  to  this  system,  however,  was  the 
need  to  constantly  maintain  the  publication  library.  Maintaining  the  system  took  a 
significant  amount  of  time,  and  publications  were  often  missing  change  notices,  updates, 
or  complete  documents. 

4.  Electronic  Policy  Document  System 

The  advent  of  the  internet  brought  forth  a  new  and  more  useful  medium  through 
which  the  Department  of  Defense  (DoD)  could  distribute  its  policies.  In  a  policy 
statement  issued  in  1998,  the  DoD  Web  Site  Administration  Policy  and  Procedures 
proclaimed,  “It  is  the  policy  of  the  DoD  that... using  the  World  Wide  Web  is  strongly 
encouraged  in  that  it  provides  the  DoD  with  a  powerful  tool  to  convey  information 
quickly  and  efficiently  on  a  broad  range  of  topics  relating  to  its  activities,  objectives, 
policies  and  programs.”6 


5  Marine  Corps  Order  P4400.150E 

^Department  of  Defense,  Web  Site  Administration  Policies  and  Procedures,  Part  1.  4.1  (1998). 
http://www.defenselink.mil/webmasters/policv/dod  web  policy  12071998  with  amendments  and  correc 

tions.html  (accessed  December  6,  2007). 
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The  Marine  Corps  embraced  this  move  toward  paperless  policy  documents  in 
1997.  It  was  then  that  the  Corps  began  revising  its  printing  and  publications  regulations 
to  take  advantage  of  new  technologies,  and  to  comply  with  the  Paperwork  Reduction  Act 
and  the  Assistant  Commandant  of  the  Marine  Corps  (ACMC’s)  Paperwork  Reduction 
Initiative.7  In  1998  the  regulation  was  updated  to  authorize  electronic  publication 
libraries  rather  than  printed  libraries.  And,  in  2001,  a  further  update  announced,  “The 
Marine  Corps  homepage  at  http://www.usmc.mil  is  the  official  Marine  Corps  web  page  to 
access  Marine  Corps  directives  online.”8  These  announcements  authorized  electronic 
access  alone  as  sufficient  to  meet  the  requirement  for  on-hand  publications  libraries  as 
long  the  documents  were  kept  up-to-date.  The  policies  further  stipulated  that  the  internet 
could  be  used  as  a  source  as  long  as  the  digitized  fdes  were  downloaded  and  kept 
locally.9 

Many  units  have  since  abandoned  maintenance  of  a  paper-based  library  in  favor 
of  an  electronic  library.  This  migration  is  in  keeping  with  the  DoD  and  Marine  Corps 
trend  of  converting  (or  originating)  the  majority  of  publications  and  documents  to  digital 
format  and  making  them  available  for  use  electronically.  Steve  Sherman,  Director  of  the 
Document  Automation  and  Production  Service  (DAPS),10  acknowledges  this  trend  on  the 
DAPS  website,  stating  “online  document  services  are  now  the  largest  part  of  our 
business.  They  have  increased  by  1 14  percent  in  the  past  seven  years.”11 

5.  Business  Case  for  Usability 

Regardless  of  the  medium,  the  goal  of  policy  is  to  ensure  compliance  with 
procedure,  and  supply  officers  use  policy  documents  as  a  primary  resource  to  achieve  that 

7  Marine  Corps  Order  P5600.31G  w/  Change  1-3,  Printing  and  Publications  Regulations,  Background 
(2001).  http://www.usmc.mil/directiv.nsf/gam?OpenView&Count=3QOO  (accessed  October  5,  2007). 

8  Ibid.,  part  3113. 

9  Ibid.,  part  3206.1. 

10  DAPS  is  a  division  of  the  Defense  Logistics  Agency  charged  with  providing  effective  and  efficient 
document  services  support  to  the  DoD  components.  Department  of  Defense  Instruction  5330.03,  Document 
Automation  and  Production  Service  (February  2006). 

1 1  Document  Automated  Printing  Service  website, 
http://www.dans.dla.mil/DAPS  MORE  THAN  A  PRINTER.asp  (accessed  September  26,  2007). 
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goal.  As  previously  noted,  supply  officers  have  traditionally  used  a  paper-based 
document  system  that  afforded  them  the  opportunity  to  personalize  documents,  to  have 
them  readily  available,  and  to  search  and  navigate  them  in  familiar  ways.  Today,  supply 
officers  have  an  electronic  capability  to  achieve  their  goals.  However,  using  an 
electronic  system  may  be  challenging  if  the  system  does  not  properly  address  the 
differing  aspects  of  electronic  and  paper-based  documents. 

Significant  research  exists  regarding  the  differences  in  electronic  and  paper-based 
documents.  In  a  study  of  hypertext  documents,  for  example,  researchers  showed  that  if 
electronic  documents  merely  imitate  paper,  the  paper  documents  will  be  preferred.  The 
research  also  showed  that  readers  who  feel  disoriented,  worry  they  are  overlooking 
crucial  information,  or  cannot  use  the  search  mechanism  will  either  abandon  the 
electronic  document  for  paper,  or  abandon  the  task  altogether.12  Other  research  argues 
that  electronic  documents  “must  be  at  least  as  easy  to  use  as  a  printed  book.”13 
Therefore,  merely  scanning  paper  documents  onto  the  web  may  not  fully  maximize  the 
intent  of  the  DoD  policy  of  conveying  information  quickly  and  efficiently,  and  users  may 
opt  instead  for  the  old  paper-based  system. 

A  more  effective,  efficient,  and  satisfying  electronic  document  system  should 
incorporate  well-designed  documents  and  institute  advantages  of  online  documents 
available  via  the  internet.  For  example,  one  advantage  is  keyword  searching,  which 
enables  an  officer  to  find  a  specific  area  of  interest  in  a  large  document,  or  in  a  collection 
of  documents,  within  seconds.  Another  advantage  of  online  documents  is  the  ability  to 
hyperlink  to  related  areas  within  or  between  documents,  which  may  further  enhance  the 
officer’s  ability  to  access  cross-references  more  efficiently.  These  features  and  others  of 


12  Florence  M.  Fillion  and  Craig  D.  B.  Boyle,  “Important  issues  in  hypertext  documentation 
usability,”  In  Proceedings  of  the  9th  Annual  international  Conference  on  Systems  Documentation 
(Chicago,  Illinois,  1991).  SIGDOC  '91.  ACM,  New  York,  NY,  59-66.  DOI= 
http://doi.acm.org/10.1145/122778.122788. 

13  Laurie  Kantner,  Roberta  Shroyer,  and  Stephanie  Rosenbaum,  "Structured  heuristic  evaluation  of 
online  documentation,"  In  Professional  Communication  Conference,  (2002).  IPCC  2002.  Proceedings. 
IEEE  International,  331-342. 

http://ieeexplore.ieee.Org/iel5/8 11 8/22469/0 1049 1 15, pdf?isnumber=22469n=STD&amumber=1049115&a 

rnumber=10491 15&arSt=+331&ared=+342&arAuthor=Kantner%2C+L.%3B+Shroyer%2C+R.%3B+Rose 

nbaum%2C+S. 
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an  electronic  document  system  can  reduce  the  burden  of  policy  retrieval,  potentially 
resulting  in  a  higher  level  of  policy  compliance. 

Additionally,  these  features  need  to  be  well-designed  and  incorporated  in  an 
overall  system  that  aids  the  supply  officer  in  finding  policy  infonnation  efficiently.  For 
example,  an  ineffective  search  engine  that  provides  unexpected  results  may  be  far  less 
effective  than  a  paper-based  system:  in  short,  the  features  must  support  the  specific  needs 
of  the  user.  If  the  Marine  Corps  intends  for  the  supply  community  to  reap  benefits  from, 
and  fully  embrace,  an  electronic  supply  policy  document  system,  the  system  must  be  one 
that  works  effectively  and  efficiently.  An  exploratory  usability  study  is  one  method  to 
assess  the  effectiveness  of  a  system  and  to  provide  a  basis  for  recommended  changes. 

6.  Overview  of  Usability  Research 

The  International  Organization  for  Standardization  (ISO  9241-11),  defines 
usability  as: 

The  extent  to  which  a  product  can  be  used  by  specified  users  to  achieve 

specified  goals  with  effectiveness,  efficiency  and  satisfaction  in  a 

specified  context  of  use.14 

There  are  three  critical  issues  embedded  in  this  definition: 

1 .  Usability  tests  are  focused  on  a  product 

2.  Usability  is  defined  and  measured  along  three  axes 

3.  Usability  tests  have  specific  users,  tasks,  and  contexts 


First,  the  goal  of  usability  tests  is  to  improve  the  product,  not  explain  or  predict 
phenomena  through  inference  or  extrapolation  of  data.  Usability  practitioners  Dumas  and 
Redish  define  this  goal  as  follows: 


14 


International  Organization  for  Standardization,  Guidance  on  Usability,  ISO  9241-11  (1998). 
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The  primary  goal  of  a  usability  test  is  to  improve  the  usability  of  the 
product  that  is  being  tested... This  characteristic  distinguishes  a  usability 
test  from  a  research  study,  in  which  the  goal  is  to  investigate  the  existence 
of  some  phenomenon.15 

Second,  usability  is  defined  and  measured  along  three  specific  criteria: 
effectiveness,  efficiency,  and  satisfaction.  These  elements  are  defined  as  follows: 


•  Effectiveness:  The  user’s  ability  to  achieve  specific  goals  in  the 
environment 

•  Efficiency:  The  resources  used  (time,  money,  and  mental  effort)  when 
performing  a  system-supported  task 

•  Satisfaction:  The  user’s  comfort  level  and  acceptance  of  the  system 
overall16 


These  three  factors  can  be  measured  in  a  variety  of  ways  that  may  include,  for 
example,  tracking  of  errors  encountered  [effectiveness],  number  of  mouse  clicks  to  a 
defined  destination  [efficiency],  time  spent  on  a  task  [efficiency],  confusion/frustration 
observed  [satisfaction],  and  many  others.17  (The  specific  measures  that  I  examined  for 
this  study,  and  their  accompanying  metrics,  are  discussed  in  more  detail  in  Methods, 
Chapter  II). 

Finally,  a  usability  test  is  distinguished  by  its  focus  on  specific  users  doing 
specific  tasks  in  specific  contexts  of  use.  When  designing  usability  tests,  the  usability 
practitioner  must  define  who  the  users  are,  what  common  tasks  are  to  be  investigated, 
under  what  conditions  the  test  will  occur,  and  how  usability  will  be  measured.  For 
example,  in  this  study,  I  am  interested  in  how  supply  officers,  in  their  daily  work,  locate 

15  Joseph  S.  Dumas  and  Janice  C.  Redish,  A  Practical  Guide  to  Usability’  Testing  (Norwood,  NJ: 

Ablex  Publishing,  1993),  22. 

16  Ibid. 

17  Ibid,  185. 
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electronic  policy  documents.  I  therefore  selected  active  supply  officers,  situated  the  test 
at  the  participants’  work  space,  and  asked  them  to  do  common  representative  tasks  (A 
more  detailed  description  of  criteria  used  to  select  appropriate  participants,  tasks,  and  key 
interactions  is  discussed  in  Methods,  Chapter  II). 

There  is  some  debate  in  the  literature  about  how  many  users  are  necessary  to 
derive  valid  results.  However,  studies  have  shown  that  five  users  can  discover  80%  of 
usability  problems.18  Usability  expert  Jakob  Nielsen  pioneered  a  1989  study  that 
established  a  precedent  for  the  required  number  of  users  for  what  he  termed  a  “discount” 
usability  study.  He  argued  that  “elaborate  usability  tests  are  a  waste  of  resources.  The 
best  results  come  from  testing  no  more  than  five  users  and  running  as  many  small  tests  as 
you  can  afford.”19  He  argues  that  after  six  users,  there  is  a  clearly  diminished  return  for 
number  of  problems  detected.  As  Figure  1  (below)  indicates,  there  is  a  negligible  gain  of 
problem  identification  between  6  and  9  users,  and  almost  no  gain  between  9  and  15  users. 

More  recently,  some  research  has  challenged  the  assertion  that  five  participants 
are  enough  for  a  sufficient  study.  In  2001,  Jared  Spool’s  report  contradicted  the  model 
after  conducting  a  series  of  website  tests  which  only  indicated  identification  of  35%  of 
problems  after  five  users,  and  some  serious  usability  problems  were  not  identified  until 
after  the  thirteenth  or  fifteenth  user.20 

Despite  these  challenges,  however,  the  purpose  of  usability  testing  must  be  taken 
into  consideration — usability  is  about  improving  the  design  of  a  product.  Even  Spool 
acknowledges  that  any  number  of  users  can  help  uncover  issues:  “For  whatever  value  of 
‘N’  [number  of  users]  they  [design  teams]  choose,  ‘N’  users  will  always  be  better  than 


18  Jakob  Nielsen,  “Usability  engineering  at  a  discount.”  In  Proceedings  of  the  Third  international 
Conference  on  Human-Computer  interaction  on  Designing  and  Using  Human-Computer  interfaces  and 
Knowledge  Based  Systems  (2nd  Ed.)  (Boston,  Massachusettss).  G.  Salvendy  andM.  J.  Smith,  Eds.  Elsevier 
Science,  New  York,  NY,  394-401. 

19  Jakob  Nielsen’s  usability  Web  site,  http://www.useit.com/alertbox/2000Q3 19.html  (accessed 
October  10,  2007). 

20  Jared  Spool  and  Will  Schroeder,  “Testing  web  sites:  five  users  is  nowhere  near  enough.”  In  CHI  '01 
Extended  Abstracts  on  Human  Factors  in  Computing  Systems  (Seattle,  Washington,  March  31  -  April  05, 
2001).  CHI  '01.  ACM,  New  York,  NY,  285-286.  DOI=  http://doi.acm.org/10. 1 145/634067.634236 
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zero.”21  Other  types  of  studies  may  net  greater  findings,  but  can  be  very  costly.  And,  as 
Neilsen  argues,  a  considerable  number  of  problems  can  be  detected  even  with  only  a  few 
participants.  No  study  will  be  “perfect,”  he  noted,  nor  will  it  “discover  everything  that’s 
possible  to  know  about  the  design,  but  we  accept  this  trade-off  in  return  for  having  more 

99 

iterations  in  the  design  process.” 


Number  of  Test  Users 
Figure  1.  Usability  Findings  by  Number  of  Users23 


In  this  usability  study,  there  is  no  expectation  that  all  the  usability  problems  will 
be  uncovered.  However,  as  Neilsen  points  out,  we  can  still  expect  to  gain  valuable 
insights  from  which  we  can  derive  recommendations. 


21  Nigel  Bevan  and  others,  “The  ‘magic  number  5’:  is  it  enough  for  web  testing?”  In  CHI  '03 
Extended  Abstracts  on  Human  Factors  in  Computing  Systems  (Ft.  Lauderdale,  Florida,  USA,  April  05  -  10, 
2003).  CHI  '03.  ACM,  New  York,  NY,  698-699.  DOI=  http://doi.acm.org/10. 1 145/765891.765936 

22  Ibid. 

23  See  note  19. 
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7. 


Research  Questions 


For  this  study,  I  identified  three  specific  research  questions  related  to  how  well 
supply  officers  could  retrieve  online  policy  information: 

1 .  Can  users  easily  and  efficiently  locate  financial  management  and  property 
control  regulations? 

2.  Can  users  easily  and  efficiently  locate  all  necessary  information  to  answer 
work-related  questions? 

3.  How  do  users  look  for  information? 

8.  Preview  of  Project 

Following  this  introductory  chapter,  the  rest  of  the  project  is  divided  into  three 
additional  chapters.  Chapter  II  presents  the  methods  I  used  in  preparing  for  and 
conducting  the  usability  test.  Chapter  III  presents  the  findings  from  the  study,  including 
direct  quotes  from  participants  as  well  as  observations  of  user  behavior.  Chapter  IV 
provides  recommendations  for  changes  to  the  system  based  on  the  findings  derived  from 
testing  and  user  feedback. 
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II.  METHODS  AND  PROTOCOLS 


This  chapter  details  the  procedures  used  to  conduct  the  usability  test.  In  the 
Methods  section,  I: 

•  Provide  an  overview  of  the  test 

•  Describe  the  test  participants 

•  Describe  users’  key  tasks 

•  Present  the  underlying  research  questions 

•  Present  the  test  scenarios 

•  Present  the  metrics  and  protocols  used  for  analysis 

In  the  Protocols  section,  I  describe  specific  approaches  used  in  collecting  and 
analyzing  the  survey  and  verbal  data. 

A.  METHODS 

1.  Overview 

The  usability  test  involved  five  participants  who  would  regularly  use  the  system 
to  retrieve  policy  documents  online  in  the  daily  conduct  of  their  work.  For  the  purposes 
of  this  study,  I  defined  the  online  system  as  “the  collection  of  official  policy  source 
documents  and  any  means  used  to  electronically  retrieve  those  documents.”  This 
definition  allowed  me  to  explore  the  user’s  natural  approach  to  locating  and  searching 
electronic  policy  documents. 

I  chose  participants  based  on  preliminary  research  of  users  of  the  documents;  I 
reviewed  demographic  data  for  the  Marine  Corps  and  conducted  a  preliminary  survey. 
The  preliminary  survey  focused  on  identifying  common  information  retrieval  tasks  and 
underlying  user  goals. 
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During  the  test,  participants  were  each  given  two  scenarios  that  required  them  to 
navigate  and  search  online  policy  documents.  While  executing  specific  tasks, 
participants  were  asked  to  “think  aloud”  while  the  researcher  directly  observed  and 
video-taped  the  sessions.  Upon  completion  of  the  specified  tasks,  the  participants 
completed  a  15 -minute  interview  followed  by  a  short,  10-question,  satisfaction  interview 
questionnaire  known  as  the  System  Usability  Scale  (SUS)  (See  Appendix  A).  The 
interview  and  SUS  explored  participants’  experience  with  and  evaluation  of  the  system. 

2.  Preliminary  User  Survey 

Prior  to  designing  the  usability  test,  I  wanted  to  understand  the  key  drivers  for 
information  retrieval.  To  help  me  identify  the  appropriate  participants  and  tasks,  I 
created  a  short  survey  instrument  (see  Appendix  B)  that  identified  common  policy  areas 
and  specific  tasks  that  Marine  Corps  supply  officers  would  likely  encounter.  For 
example,  I  selected  key  areas  such  as  property  control  or  requisition  management. 
Within  each  area,  I  also  provided  a  selection  of  tasks  that  would  be  appropriate  to  that 
area,  such  as  sub-custody  receipting  (CMRs)  or  inventory  would  be  for  the  property 
control  area.  Drawing  from  the  Total  Force  Data  Warehouse  (TFDW)  database24,  I 
identified  258  potential  participants  and  contacted  them  via  email.  Of  the  258  number 
contacted,  109  responded.  The  survey  asked  respondents  to  select  the  policy  area  they 
most  frequently  consulted  and  the  primary  tasks  within  those  policy  areas  that  they  most 
frequently  conducted.  Additionally,  I  asked  them  to  identify  the  three  most  common 
reasons  for  using  electronic  publications.  Finally,  I  used  the  survey  as  a  recruiting  tool: 
the  final  question  asked  if  they  were  willing  to  be  contacted  to  be  a  participant  in  the 
study.  Results  are  summarized  in  Table  1,  below: 


24 


TFDW  accessed  via  Marine  Online  website,  http://mol.usmc.mil  (accessed  September  1,  2007). 
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Table  1.  Summary  of  results  from  preliminary  user  survey 


Response 

Three  Most  Common  Reasons  for  Consulting  Policy  Documents 

59.8% 

Advise  Commander 

57.8% 

Prove/Justify  Procedures  to  an  Interested  Party 

53.9% 

Verify  Personal  Action/Procedure  Complies  with  Reference 

Response 

%  Time  Spent  Seeking  Policy  Information  in  Electronic  Documents 

41.2% 

10%  -  20% 

22.7% 

Less  than  10% 

20.6% 

20%  -  30% 

Response 

Area  and  (Task  within  that  area)  Requiring  the  Most  Time 

30.4%  (64.5%) 

Financial  Management  (Government  Commercial  Purchase  Card) 

29.4%  (44.8%) 

Property  Control  (CMR) 

I  used  the  results  of  this  preliminary  survey  to  select  test  participants  and  to 
choose  test  scenarios  and  tasks. 

3.  Participants 

In  usability  tests,  it  is  important  to  be  sure  that  our  test  participants  align  with  our 
business  and  practical  goals.  In  this  current  study  of  electronic  supply  documents, 
possible  user  populations  included  supply  personnel,  government  contractors,  legislative 
bodies,  commanders,  and  many  others.  However,  for  this  usability  test,  I  focused 
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specifically  on  intermediate-level  officer  leadership  (company  grade  officers)  within  the 
Marine  Corps.  I  selected  this  representative  user  group  for  four  primary  reasons: 

1 .  They  have  a  large  impact  on  the  organization 

2.  Their  characteristics  are  typical  of  the  targeted  user  base 

3.  They  have  varying  degrees  of  familiarity  with  the  interface 

4.  They  were  readily  and  locally  available 


Additionally,  based  on  the  rationale  provided  by  Neilsen’s  standard-setting 
research  showing  that  nearly  80%  of  usability  problems  can  be  found  by  testing  only  five 
users,25 1  targeted  five  users  for  this  test. 

a)  Supply  Officers  ’  Impact  on  the  Organization 

When  selecting  test  participants,  I  wanted  to  ensure  that  the  participants 
were  key  users  of  the  system.  When  looking  at  the  various  roles  associated  with  the  use 
of  supply  policy  documents,  supply  officers  stood  out  as  key  business  users.  As  advisors 
to  the  Commander,  trainers  of  junior  officers,  and  supervisors  of  enlisted  men,  supply 
officers  have  a  large  impact  on  their  command.  First,  supply  officers  are  directly  charged 
with  advising  and  recommending  corrective  procedures  based  on  policies,  laws,  and  other 
regulatory  guidance  contained  in  supply  policy  documents.  Second,  the  Marine  Corps 
mandates  that  supply  officers  establish  an  “ongoing  and  effective  training  program”  on 
policies  and  procedures.'  Even  though  actual  training  is  often  conducted  by  subordinate 
leaders,  the  overall  responsibility  for  creating  the  training  is  explicitly  that  of  the  supply 
officer.  Finally,  given  that  there  is  approximately  one  officer  for  every  ten  enlisted 


25  See  note  18  above. 

26  See  note  4  above. 

27  MCO  P4400.150E,  Section  1011. 
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9R 

Marines,"  the  supply  officer  has  significant  supervisory  and  oversight  responsibilities 
related  to  policy  understanding  and  compliance. 

b)  Supply  Officers  are  Typical  of  Targeted  User  Base 

In  usability  testing,  it  is  important  to  select  your  test  participants  based  on 
the  alignment  of  their  personal  characteristics  with  the  key  user  population.  Key 
characteristics  to  look  at  include  personal  demographics  and  skill  levels.  As  part  of  our 
recruiting  process,  we  verbally  verified  that  participants  met  the  desired  characteristics 
prior  to  scheduling  them.  In  addition,  we  verified  their  characteristics  prior  to 
administering  the  test. 

For  this  study,  I  selected  participants  that  would  closely  match  the 
demographics  of  supply  officers.  Marine  Corps  officers  are  largely  homogenous  in  terms 
of  education,  rank,  and  age.  In  addition,  occupational  assignments  within  the  Marine 
Corps  are  not  tied  to  skills — all  supply  officers  receive  formal  supply  training  at  the 
Ground  Supply  Officer  Course  (GSOC).  Therefore,  we  selected  participants  based  on  the 
following  Marine  Corps  demographic  trends: 

•  90%  hold  a  BA  degree  or  higher  29 

•  55%  of  total  supply  officers  are  captains  or  first  lieutenants 

•  47%  are  between  the  ages  of  26  and  3530 

•  Captains  predominantly  have  4-10  years  military  experience  and  first 

lieutenants  have  2-4  years  experience31 


Headquarters,  United  States  Marine  Corps,  2007  Marine  Corps  Almanac,  Headquarters  Marine 
Corps  website  (2007).  http://hqinet001.hqmc.usmc.mil/p&r/concepts/2007/CHPT5.htm  (accessed 
September  18,  2007). 

29  Marine  Corps  Community  Services,  Marine  Corps  Community  Services  Demographics  Update 
(June  2007).  http://www.usmc-mccs.org/aboutmccs/downloads/pom/Demographics%20Update.pdf 
(accessed  October  12,  2007).  The  data  for  the  report  is  extracted  from  the  Marine  Corps  Total  Force  Data 
Warehouse. 

311  See  note  28,  above. 

31  Department  ofDefense  Instruction  1320.13,  Commissioned  Officer  Promotion  Reports  (COPRs) 
and  Procedures,  (June  1996).  http://wwwQ.dtic,mil/whs/directives/corres/html/ 1320 13.htm  (accessed 
October  12,  2007). 
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c)  Supply  Officers  Have  Varying  Familiarity  with  the  Interface 


Within  my  five  participants,  I  recruited  officers  who  were  both  more 
familiar  and  less  familiar  with  retrieving  policy  information  online  (see  Table  2,  below). 
These  participants  represented  the  larger  user  population  whose  responses  in  the 
preliminary  survey  indicated  similar  variance  in  familiarity.  Specifically,  41.2%  of 
respondents  indicated  that  they  spent  between  10%  and  20%  of  their  time  looking  for 
information  in  electronic  publications,  and  another  22.7%  indicated  that  they  spend  less 
than  10%  of  their  time  doing  so.  Therefore,  selecting  participants  that  had  different 
levels  of  familiarity  with  the  system  gave  a  better  approximation  of  the  actual  user  group 
as  a  whole. 

More  junior  officers  (second  lieutenants)  were  excluded  from  the  user 
population  since  their  lower  job  experience  level  may  have  strongly  and  negatively 
impacted  their  performance.  However,  these  more  novice  users  may  be  an  important 
secondary  audience  to  consider  in  future  studies. 

d)  Participant  A  vailability 

Finally,  while  1  tried  to  recruit  representative  participants,  the  short  time 
frame  for  this  study,  and  the  resources  involved  in  soliciting  from  a  larger  participant 
pool,  meant  that  1  had  to  limit  my  participant  base  to  California.  All  participants  were 
selected  from  Marine  Corps  Base,  Camp  Pendleton  due  to  its  proximity  and  ready  access 
to  users.  However,  to  ensure  that  our  participants  were  appropriate,  we  included  a  brief 
pre-test  questionnaire  (see  Appendix  C)  to  verify  that  their  characteristics  were  consistent 
with  the  target  user  profile  derived  from  Marine  Corps  demographics:  Captains  and  First 
Lieutenants  between  the  ages  of  26  and  35  with  no  less  than  two  years  of  service  and  a 
minimum  four-year  college  degree.  Table  2  summarizes  the  actual  characteristics  of  our 
five  test  participants. 
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Table  2.  Pre-Test  Survey  Results 


Stats  from  Pre-Test  Questionnaire 


Age  Group 

Age  26-35 

All  Users 

100% 

Internet  familiarity 

Very  familiar 

User  1,4 

40% 

Familiar 

User  2,3,5 

60% 

Time  spent  on  pubs 

0-10% 

User  1,4,5 

60% 

1 0%-20% 

User  3 

20% 

30%-50% 

User  2 

20% 

How  long  in  supply? 

yrs  2-4 

User  4,5 

40% 

yrs  4-10 

User  1,2,3 

60% 

Education 

BAS  degree 

User  2, 3, 4, 5 

80% 

graduate  degree 

User  1 

20% 

4.  Selecting  Appropriate  Tasks 

Tasks  conducted  during  a  usability  test  must  represent  tasks  that  the  defined  users 
would  realistically  be  expected  to  perfonn;  this  requirement  necessitates  an 
understanding  of  users’  jobs  and  tasks.32  The  results  of  the  preliminary  survey  (see  Table 
1,  page  15)  provided  valuable  insights  and  helped  me  to  better  understand  user  tasks. 

Driven  by  these  survey  responses  and  my  own  experience  with  practical  situations 
in  those  areas,  I  created  two  scenarios  based  on  the  most  common  tasks  selected  by 
respondents33  (a  synopsis  of  the  scenarios  is  provided  later  in  this  chapter). 

5.  Targeting  Components 

A  usability  test,  depending  on  its  goals,  targets  certain  components  of  the  overall 
system.  Typical  components  include  help  systems,  navigation  systems,  page  layouts,  and 
others.  However,  testing  every  component  of  a  system  can  be  cost-prohibitive  and  time- 

32  Dumas  and  Redish,  A  Practical  Guide  to  Usability  Testing,  23. 

33  Test  results  indicated  that  users  found  the  selected  tasks  to  be  relevant  and  realistic  of  those  they 
would  conduct  on  the  job  (See  chapter  III). 
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consuming.  In  this  study,  I  was  interested  in  understanding  how  users  “naturally”  use  the 
system  to  locate  and  retrieve  information;  in  other  words,  users  needed  to  be  free  to 
navigate  and  search  as  they  would  normally.  Therefore,  for  this  test,  I  selected 
components  which  are  key  to  locating  information — information  navigation  and  search. 

a)  Information  Navigation 

The  policy  documents,  as  previously  indicated,  come  from  multiple 
sources,  in  a  variety  of  policy  document  types,  and  have  frequent  changes,  supplements, 
and  cancellations.  Additionally,  many  Web  sources  make  internal  references  and  cross- 
references  to  other  policy  documents  which  may  or  may  not  be  located  under  the  same 
source.  This  system  complexity  may  impede  the  users’  ability  to  find  the  documents  they 

need,  and  to  collate  the  information  in  an  effective  and  efficient  manner.  To  understand 
navigation  difficulties  better,  I  focused  on  instances  where  a  user  cannot  accomplish  the 
following: 

•  Identify  the  correct  location 

•  Return  to  a  previous  location 

•  Find  information  believed  to  exist  somewhere,  or 

•  Remember  key  points  learned34 

If  a  navigational  aid  is  effective,  it  should  assist  the  user  in  finding 
information  effectively  and  efficiently. 

b)  Search 

A  primary  benefit  of  electronic  resources  is  that  systems  can  improve  how 
users  can  search  for  infonnation,  which  can  positively  impact  effectiveness  and 
efficiency.  Keyword  searching,  hyperlinking,  and  other  methods  have  brought  new 
access  capabilities  when  compared  with  paper-based  systems.  Search  mechanisms  are 

34  Deborah  M.  Edwards  and  Lynda  Hardman,  “Lost  in  hyperspace:  cognitive  mapping  and  navigation 
in  a  hypertext  environment”  In  Hypertext:  theory’  into  Practice ,  R.  McAleese,  Ed.  Intellect  Books,  (Exeter, 
UK,  1999),  90-105. 
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particularly  valuable  with  regard  to  policy  documents  where  the  user’s  primary  purpose  is 
to  retrieve  specific  policy  information  rather  than  simply  browsing  or  reading  for 
enjoyment.  In  this  study,  I  targeted  the  component  of  search  mechanisms  because  of  its 
particular  importance  to  my  users’  key  tasks. 

B.  RESEARCH  QUESTIONS  AND  USABILITY  MEASUREMENTS 

Scenario  tests  should  be  based  on  realistic  scenarios  that  drive  to  specific  research 
questions.  For  this  study,  I  identified  three  specific  research  questions  that  focused  on 
basic  information  retrieval  (question  1)  and  on  the  larger  issues  of  goal  (question  2)  and 
process  (question  3): 

1 .  Can  users  easily  and  efficiently  locate  financial  management  and  property 
control  regulations? 

2.  Can  users  easily  and  efficiently  locate  all  necessary  information  to  answer 
work-related  questions? 

3.  How  do  users  look  for  information? 

1.  Usability  Measurements 

Each  of  the  above  research  questions  were  broken  down  into  key  usability 
elements  around  the  issues  of  navigation  and  search.  I  have  also  identified  the  primary 
modes  of  answering  these  research  questions  (indicated  parenthetically). 

a)  Issue  1  —  Effectiveness 

Issue  1.1  Do  the  navigation  and  search  tools  effectively  aid  users  in  locating 
a  Marine  Corps  policy  document  that  will  help  them  achieve  their 
goals?  (task  one  failure  rates  -  property  control  scenario) 
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Issue  1.2 


Do  the  navigation  and  search  tools  effectively  aid  users  in  locating 
policy  documents  that  originate  from  other  sources?  (task  three 
failure  rates  -  financial  management  scenario) 

Issue  1.3  Do  search  tools  and  navigational  design  within  documents  provide 
an  effective  means  for  users  to  locate  critical  information  related  to 
their  goals?  (task  two  and  task  four  failure  rates,  verbal  data) 

Issue  1.4  Degree  to  which  users  believe  the  system  supports  their  work 
(subjective  scale  question) 

b)  Issue  2  —  Efficiency 

Do  navigation  design  and  search  tools  assist  users  by  enabling  easy 
and  efficient  location  of  policy  documents?  (number  of  methods 
used/paths  followed,  failed  keyword  searches,  verbal  data) 

Do  navigation  design  and  search  tools  assist  users  by  enabling  easy 
and  efficient  searching  for  information  within  documents?  (number 
of  methods  used/paths  followed,  failed  keyword  searches,  verbal 
data) 

c)  Issue  3  —  Satisfaction 

Issue  3.1  Overall  user  confidence  in  the  system  (subjective  scale  questions, 
System  Usability  Score  (SUS),  verbal  data) 

2.  Summary 

The  following  summary  table  provides  a  graphic  illustration  of  how  the  scenarios 
relate  to  the  areas  of  concern  (issues)  and  the  components  for  which  I  desire  to  gain 
insight  through  the  testing.  Additionally,  the  methods  of  observation  for  each  scenario 
are  also  provided: 


Issue  2.1 


Issue  2.2 
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Table  3.  Project  Overview 


Task 

Scenario 

Issue 

Components 

Observation 

1 

•  Electronically  locate 
supply  policy 
documents  to  guide 
you  in  accounting  for 
SL-3  items 

•  Easily  Locate  Marine 
Corns  documents 

•  Currency/source 
reliability 

•  Satisfaction 

•  Navigation  Aids 

•  Search  tools 

•  Task  failure  rates 

•  Survey  response 

•  Verbal 

•  SUS 

2 

•  Use  electronic  supply 
policy  documents  to 
determine  authority  to 
establish  quantity,  and 
procedures  for 
accounting  for  UURI 
SL-3  items 

•  Locate  critical 
information 

•  Easily  search  critical 
information 

•  Satisfaction 

•  Navigation  Aids 

•  Search  tools 

•  Task  failure  rates 

•  Methods  used 

•  Failed  Searches 

•  Verbal 

3 

•  Electronically  locate 
governing  GCPC 
policy  documents 

•  Easily  locate 
documents  from 
another  source 

•  Currencv/source 
reliability 

•  Satisfaction 

•  Navigation  Aids 

•  Search  tools 

•  Task  failure  rates 

•  Survey  response 

•  Verbal 

•  SUS 

4 

•  Use  electronic  GCPC 
documents  to 
determine  authority  to 
purchase 

•  Locate  critical 
information 

•  Easily  search  critical 
information 

•  Satisfaction 

•  Navigation  Aids 

•  Search  tools 

•  Task  failure  rates 

•  Methods  used 

•  Failed  Searches 

•  Verbal 

Each  of  my  research  questions,  along  with  the  targeted  tasks  and  components, 
drove  the  design  of  the  test  scenarios.  My  goal  was  to  create  test  scenarios  and  tasks  that 
would  drive  users  to  suspected  usability  problem  areas  and  provide  the  basis  for 
measurements  of  usability.35 


35  Dumas  and  Redish,  A  Practical  Guide  to  Usability  Testing^llb. 
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3. 


Scenarios  and  Tasks 


The  following  section  provides  a  synopsis  of  the  scenarios  and  tasks  given  to 
participants.  The  researcher  created  the  tasks  in  accordance  with  guidance  from  A 
Practical  Guide  to  Usability  Testing  36  that  cites  four  criteria  for  selecting  tasks: 

•  Tasks  that  probe  potential  usability  problems 

•  Tasks  suggested  from  your  concerns  and  experience 

•  Tasks  derived  from  other  criteria 

•  Tasks  that  users  will  do  with  the  product 

The  preliminary  survey  results  indicated  that  the  two  tasks  most  commonly 
conducted  were  in  the  areas  of  property  control  and  financial  management.  So, 
participants  were  given  two  scenarios,  one  in  each  of  the  two  areas.  After  the  test, 
participants  confirmed  that  the  scenarios  were  realistic.  As  one  participant  put  it,  “these 
are  great  scenarios. .  .because  I’ve  been  there.”  Another  participant  noted,  “It’s  funny, 
I’ve  actually  seen  stuff  like  this  before,”  and  still  another  echoed  that,  “I’ve  had  personal 
experience  with  something  similar.”  The  complete  details  of  the  scenarios  and  tasks  can 
be  found  in  the  test  plan  in  Appendix  D. 

a)  Property  Control  Scenario 

The  property  control  scenario  required  the  participant  to  investigate  proper 
accountability  procedures  regarding  “Using  Unit  Responsibility  Item”  (UURI).  The 
specific  instructions  required  the  participant  to  advise  the  commander  as  to  whether  or 
not  specific  items  are  to  be  recorded  on  supply  property  records  and,  if  so,  who  would  be 
authorized  to  establish  an  allowance  for  such  an  item. 

Task  One.  The  first  task  under  this  scenario  required  participants  to  locate 
policy  documents  they  thought  were  applicable.  Participants  may  or  may  not  know  the 
title  or  numerical  designator  of  the  policy  source  when  executing  the  task.  The  task 


36  Ibid,  160. 
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required  search  and  navigation  to  a  specific  internal  Marine  Corps  supply  procedure. 
Participants  were  given  a  five  minute  time  limit  to  complete  this  task.  The  task 
specifically  targets  the  user’s  ability  to  locate  a  particular  Marine  Corps  policy  document 
that  will  help  them  achieve  their  goals  (issues  1.1  and  2.1). 

Task  Two.  The  second  task  for  this  scenario  targets  a  more  complex 
issue — whether  or  not  users  can  use  navigation  and  search  aids  within  a  document  to 
effectively  and  efficiently  locate  and  collate  information  into  a  cohesive  response  to  a 
policy-related  question  (issues  1.3,  1.4,  2.2).  A  time  limit  of  ten  minutes  applied  to  this 
task. 


b)  Financial  Management  Scenario 

The  financial  management  scenario  required  the  participant  to  determine 
the  legality  of  a  proposed  purchase  of  un-priced  maintenance  services  for  a  General 
Services  Administration  (GSA)  vehicle  with  the  Government  Commercial  Purchase  Card 
(GCPC),  given  an  estimated  cost  of  $2600. 

Task  Three.  Task  three  falls  under  the  financial  management  scenario  and 
essentially  mirrors  task  one  in  the  preceding  scenario;  the  participant  must  locate  the 
applicable  electronic  policy  documents  within  a  five  minute  time  limit.  The 
differentiating  factor  is  that  the  GCPC  program  is  not  governed  directly  by  the  Marine 
Corps  and  thus  tests  how  users  might  look  for  policy  documents  when  the  source  is  less 
obvious.  Therefore,  this  task  targets  the  user’s  ability  to  locate  a  policy  document  that 
originates  from  a  source  other  than  the  Marine  Corps  and  that  will  help  achieve  the  user’s 
goals  (issue  1.2  and  2.1). 

Task  Four.  Task  four  targets  the  user’s  ability  to  easily  and  efficiently 

search  for  information  within  documents  to  answer  a  work-related  question.  In  this 

scenario,  participants  were  asked  to  identify  three  items:  whether  the  purchase  exceeds 

the  purchase  amount  threshold,  if  the  policy  prohibits  a  purchase  of  this  type  (un-priced 

service),  and  if  any  authority  prohibits  the  purchase  of  this  particular  item  (GSA  vehicle). 

A  time  limit  of  ten  minutes  applied  and  the  resulting  data  were  used  to  address  research 

issues  relating  to  the  effectiveness  of  search  tools  and  navigational  design  within 
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documents  (issue  1.3),  the  efficiency  of  those  navigation  and  search  tools  (issue  2.2),  and 
overall  user  confidence  in  the  system  (issue  3.1). 

4.  Metrics 

In  this  section,  I  define  metrics  that  I  used  to  evaluate  performance  including 
failure  rates,  verbal  data,  System  Usability  Scale  (SUS),  number  of  paths  followed,  and 
failed  keyword  searches. 

a)  Failure  Rates  (Task  1  -  Property  Control  Scenario) 

Task  success  was  defined  as  a  participant’s  retrieval  of  a  policy  document 
which  could  provide  the  answers  required  in  task  two  within  a  time  limit  of  five  minutes. 
Time  began  after  participants  had  read  the  scenario  and  began  to  use  the  computer.  The 
primary  policy  document  which  provides  these  answers  is  Marine  Corps  Order 
P4400.150E,  the  Marine  Corps  Consumer  Level  Supply  Policy  Manual,  located  on  the 
Marine  Corps  homepage  at  www.usmc.mil.  There  are  multiple  ways  in  which  the 
document  could  have  been  accessed.  One  possible  navigational  approach  was  to  select 
‘Publications’  from  the  main  screen  menu,  and  then  select  ‘Orders  and  Directives.’  This 
would  lead  the  user  to  another  screen  which  lists  methods  of  browsing  documents.  From 
there,  selecting  ‘MCO  “P”  Directives’  would  provide  the  entire  listing  from  which  the 
user  could  select  this  document. 

b)  Failure  Rates  (Task  2  -  Property  Control  Scenario) 

Participants  who  were  unsuccessful  in  locating  a  policy  document  during 
task  one  were  given  MCO  4400. 150E  as  a  starting  point  for  task  two.  Task  success  was 
determined  by  the  participant’s  correct  response  to  the  two  requirements  of  the  task 
within  a  ten  minute  time  limit.  A  correct  response  to  the  first  question  required 
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identification  of  the  commanding  officer  as  the  proper  authority  to  establish  an  allowance 
for  a  type  II  UURI  SL-3  item.  This  information  was  located  in  paragraph  2011c,  which 
states: 

...where  “AR”  (as  required)  is  the  stated  quantity,  the  commander  must 
establish,  in  writing,  the  authorized  quantity  to  be  held  by  the  command.37 

The  correct  response  for  item  two  was  located  in  the  next  paragraph 
201  lc(l),  which  stated: 

UURI’s  that  are  identified  as  having  a  type  1  or  2  TAMCN  [Table  of 
Authorized  Material  Control  Number]  should  be  validated  against  the 
unit’s  EAF  [Equipment  Allowance  File]  and  accounted  for  on  the  unit 
property  records.38 

c)  Failure  Rates  (Task  3  -  Financial  Management  Scenario) 

Task  success  was  defined  as  a  participant’s  retrieval  of  a  policy  document 
which  originated  from  an  official  government  source,  and  which  could  provide  the 
answers  required  in  task  four,  within  a  time  limit  of  five  minutes.  Time  began  after 
participants  had  read  the  scenario  and  began  to  use  the  computer.  The  primary  policy 
document  which  governs  the  Government- Wide  Commercial  Credit  Card  Program 
(GCPC)  within  the  United  States  Marine  corps  is  NAVSUPINST  4200.99,  issued  by  the 
Department  of  the  Navy.39  A  variety  of  other  documents  also  provide  policy  guidance 
for  the  program,  and  originate  from  official  government  sources,  including  Standard 
Operating  Procedures  (SOPs),  Guidebooks,  Internal  Operating  Procedures  (IOPs)  from 
subordinate  commands  and  many  others. 

At  the  time  of  the  test,  two  primary  options  existed  to  retrieve  an 
acceptable  policy  document.  One  option  was  the  Marine  Corps’  Contract  Management 

37  MCO  P4400.150E,  Section  201  lc. 

38  Ibid.,  section  2011c(l). 

39  Naval  Supply  Instruction  (NAVSUPINST)  4200.99,  Department  of  the  Navy  (DON)  Policies  and 
Procedures  for  the  Operation  and  Management  of  the  Government-Wide  Commercial  Purchase  Card 
Program  (GCPC),  (October  2006)  (hereafter  cited  as  NAVSUPINST  4200.99). 

http ;//hqinet001. hqmc.usmc.mil/I%26LW2/CMPG/  (accessed  October  18,  2007). 
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Process  Guide  (CMPG)  (NAVSUPINST  4200.99),  located  at 
http://hqinet00 1  .hqmc.usmc.mil/i&L/v2/CMPG/index.htm.  Additionally,  the  Marine 
Corps  Base,  Camp  Pendleton,  GCPC  program  is  a  source  for  this  information.40 

d)  Failure  Rates  ( Task  4  -  Financial  Management  Scenario) 

Participants  who  were  unable  to  locate  a  document  in  task  three  were 
given  an  electronic  pdf  file  of  NAVSUPINST  4200.99  retrieved  from  the  Marine  Corps 
CMPG  site  identified  in  paragraph  (b)  above,  as  a  starting  point  for  task  two.  Task 
success  was  determined  by  the  participant’s  determination  that  the  purchase  was 
unauthorized  by  explicitly  referencing  one  of  the  three  factors  which  would  have 
prohibited  the  purchase  and  doing  so  within  a  time  limit  of  ten  minutes.  The  three  factors 
were: 

1 .  Dollar  amount  exceeds  purchase  limit  threshold,41 

2.  An  un-priced  service  without  an  established  ceiling  price  is 
unauthorized,42  or 

3.  The  specific  exclusion  of  purchasing  services  for  GSA-leased  vehicles.43 


Additionally,  the  information  must  have  been  derived  from  an  official 
government  source  as  defined  in  paragraph  (b)  above.  There  were  three  acceptable 
solutions: 


40  Regional  Contracting  Office  -  Southwest  (Marine  Corps  Base,  Camp  Pendleton,  CA). 
http://www.rcosw.com/apporders.htm  (accessed  October  10,  2007). 

41  NAVSUPINST  4200.99,  section  1.4.b.l. 

42  Ibid.,  Section  5.19. 

43  Regional  Contracting  Office  Southwest,  Government  Commercial  Purchase  Card  (GCPC) 
Program  Internal  Operating  Procedures,  (  Marine  Corps  Base  Camp  Pendleton,  California)  (September 
2003).  http://www.rcosw.com/RCO-SW%20%20IQP.pdf  (accessed  October  18,  2007). 
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1.  Using  NAVSUPINST  4200.99  or  other  official  guide  books  which 
prohibit  un-priced  services  and  sets  limits  to  the  purchase  price. 

2.  Using  Purchase  Card  Policy  Notice  (PCPN)  7  to  identify  that  the  purchase 
exceeded  the  recently  increased  purchase  limit  established. 

3.  Using  the  Regional  Contracting  Office  -  Southwest  (RCO-SW)  website 
was  accessible  at  www.rcosw.com.  to  see  that  the  ‘repair  or  alteration  of  a 
GSA-leased  vehicle’  is  prohibited  for  card  holders  under  their  purview. 

e)  Verbal  Data 

Verbal  data  are  qualitative  measurements  collected  in  accordance  with  the 
protocols  identified  later  in  this  section.  Verbal  data,  to  include  both  think- aloud 
commentary  and  interview  responses  (discussed  at  length  in  the  Protocol  section,  below), 
were  assessed  and  attributed  to  specific  problem  areas  where  applicable. 

f)  System  Usability  Scale  (SUS) 

The  SUS  survey  tool  was  developed  in  1986  by  usability  practitioner  John 
Brooke.44  The  survey  consists  of  a  series  of  ten  questions,  each  using  a  five-point  Likert 
scale  to  rate  varying  aspects  of  usability.  Scoring  is  combined  to  provide  a  single 
measure  of  overall  usability.  To  determine  the  SUS  score,  I  used  the  standard  scoring 
process  described  by  the  designer  of  the  SUS,  resulting  in  a  usability  rating  on  a  0%  - 
100%  scale.  The  protocol  and  its  specific  scoring  mechanism  are  discussed  fully  in  the 
Protocol  section  below. 

g)  Number  of  Paths  Followed 

Two  types  of  paths  were  tracked — searching  and  navigating.  Searching 
different  keywords  within  a  single  search  technique  was  not  considered  a  different 
approach.  However,  applying  a  different  search  filter  or  different  search  engine  was 
considered  a  different  search  approach.  Additionally,  if  a  search  approach  was  used  and 
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abandoned  for  another  search  approach,  returning  to  the  original  approach  would  qualify 
as  an  additional  instance  when  calculating  the  number  of  search  paths  used. 

The  other  approach  possible  was  a  navigational  trail.  An  instance  of  a 
navigational  trail  was  recorded  for  each  “wrong”  path  of  navigation.  “Wrongness”  was 
determined  by  the  participant’s  election  to  ‘go  back’  and  begin  a  different  navigational 
trial,  or  by  simply  abandoning  the  location  for  another  method. 

In  total,  the  number  of  paths  used  consists  of  the  combined  number  of 
instances  for  both  approaches. 

h)  Failed  Keyword  Searches 

A  failed  keyword  search  was  recorded  each  time  a  participant’s  keyword 
did  not  bring  him  to  a  location  which  could  provide  success  as  identified  in  each  of  the 
task  areas. 

In  addition  to  the  above  metrics,  I  employed  several  different  protocols 
which  are  discussed  in  the  next  section. 

C.  PROTOCOLS 

Protocols  used,  in  addition  to  the  above  metrics,  include  surveys,  semi-structured 
interviewing,  and  “think-aloud”  verbalizations  as  data  sources. 

1.  System  Usability  Scale  (SUS)  and  Other  Subjective  Scale  Questions 

In  1986,  usability  practitioner  John  Brooke  developed  the  System  Usability  Scale 
(SUS),  as  a  reliable,  low  cost  scale  that  can  be  used  for  global  assessments  of  systems 
usability.45  Each  question  on  the  SUS  uses  a  five  point  Likert  scale,  soliciting  the  degree 
to  which  a  participant  ‘strongly  agrees’  or  ‘strongly  disagrees’  with  statements  regarding 


44  John  Brook,  ‘‘SUS:  A  ‘Quick  and  Dirty’  Usability  Scale.”  In  Usability  Evaluation  in  Industry’,  ed. 
Patrick  W.  Jordan  (London:  Taylor  and  Francis,  1996),  189-195. 

4^  See  note  49. 
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system  usability.  In  the  1996  book,  Usability  Evaluation  in  Industry,  the  SUS  developer 
contends  that  the  most  effective  questions  utilizing  a  Likert  scale  are  those  which  solicit 
the  most  extreme  responses: 

The  items  leading  to  the  most  extreme  responses  from  the  original  pool  [of 
50  potential  questions]  were  then  selected... items  were  selected  so  that  the 
common  response  to  half  of  them  was  strong  agreement,  and  to  the  other 
half,  strong  disagreement.  This  was  done  in  order  to  prevent  response 
biases  caused  by  respondents  not  having  to  think  about  each  statement;  by 
alternating  positive  and  negative  items,  the  respondent  has  to  read  each 
statement  and  make  an  effort  to  think  whether  they  agree  or  disagree  with 
it  46 


Due  to  the  alternating  positive  and  negative  items,  the  SUS  requires  a 
scoring  technique  that  accounts  for  the  resulting  numerical  variance  in  raw  scores.47 

Results  from  the  SUS  are  presented  in  terms  of  percentages  with  100% 
indicating  perfect  usability.  As  stated,  this  SUS  scoring  method  is  the  standard  manner 
described  by  the  test’s  designer  to  give  an  overall  assessment  of  usability.  The  SUS 
questionnaire  is  provided  in  Appendix  A. 

In  addition  to  the  SUS  questions,  participants  were  asked  to  respond  to 
three  additional  statements  using  the  same  five  point  Likert  scale.  The  responses  were 
evaluated  in  relation  to  specific  research  issues  of  effectiveness  and  were  not  included  in 
the  SUS  score  calculation.  The  additional  statements  were: 

•  I  found  the  system  supports  my  work 

•  I  was  fairly  confident  that  the  information  was  the  most  current  available 

•  I  was  fairly  confident  that  I  found  all  the  infonnation  by  using  this  system 


46  See  note  49. 

47  First,  raw  scores  are  converted  to  a  SUS  contribution  score  for  each  individual  question.  SUS 
contributions  are  calculated  differently  depending  on  whether  the  question  is  worded  positively  or 
negatively.  Questions  in  which  positive  answers  generate  high  scale  ratings  were  scored  by  subtracting  one 
from  the  scale  rating  marked.  Questions  in  which  positive  answers  generate  low  scale  ratings  were  scored 
by  subtracting  the  scale  rating  marked  from  five.  This  method  gives  equal  contribution  opportunity  for 
each  question  with  a  range  of  zero  to  four  and  thus  equates  to  a  total  forty  possible  points  from  the  ten 
question  survey.  The  total  contribution,  then,  is  multiplied  by  two  point  five  in  order  to  put  the  SUS  score 
on  a  100  point  scale. 
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2. 


Think  Aloud 


Think-aloud  protocol  is  widely  accepted  for  the  collection  of  verbal  data  in  a 
variety  of  disciplines  including  cartography,  engineering  and  reading  comprehension.48 
This  type  of  protocol  calls  for  participants  to  verbalize  their  thought  processes  while 
conducting  a  task,  which  a  researcher  can  later  analyze  to  gain  insight  into  a  user’s 
cognitive  process.  Researchers  Ericsson  and  Simon  are  accredited  with  the  core  research 
for  this  type  of  verbal  data  in  their  1980  article  “Verbal  Reports  as  Data.”49 

Usability  studies  often  rely  on  this  method  for  the  collection  of  data.  Respected 
usability  practitioner,  Joe  Dumas,  indicated  in  a  recent  review  of  the  practice  that,  “Think 
aloud  is  the  most  important  method  we  have  in  the  toolkit  of  usability  evaluation.  It 
uncovers  more  problems  than  any  other  measure.”50 

There  is  some  debate  in  the  field,  however,  over  appropriate  use  of  the  protocol, 
and  the  validity  of  data,  if  it  is  not  administered  carefully.  In  a  2000  review  of  the 
method,  authors  Boren  and  Ramey  indicate  that  practitioners  can  avoid  these  pitfalls  by 
carefully  adhering  to  four  basic  principles  of  the  original  research  by  Ericsson  and 
Simon: 

•  Participant  introspection,  inference  or  opinion  must  not  be  valued  or 
actively  elicited 

•  Give  detailed  instructions  for  thinking  aloud 

•  Remind  participants  to  think  aloud 

•  Otherwise,  do  not  intervene51 

48  Judith  Ramey  and  others,  “Does  think  aloud  work?:  how  do  we  know?”  In  CHI  '06 Extended 
Abstracts  on  Human  Factors  in  Computing  Systems  (Montreal,  Quebec,  Canada,  April  22  -  27,  2006).  CHI 
'06.  ACM,  New  York,  NY,  45-48.  DOR  http://doi.acm.org/10.1145/1125451.1125464 

49  K.  Anders  Ericsson  and  Herbert  A.  Simon,  “Verbal  Reports  as  Data”,  Psychological  Review,  87, 
no.  3  (1980):  215. 

50  See  note  48. 

5 1  Ted  Boren  and  Judith  Ramey,  "Thinking  Aloud:  Reconciling  Theory  and  Practice,"  IEEE 
Transactions  on  Professional  Communication,  43,  no.  3  (2000):  261. 
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These  principles  guided  the  protocol  application  for  this  usability  test. 
Participants  were  told  about  the  think  aloud  protocol  and  given  a  detailed  description  of 
the  process  prior  to  conducting  the  scenarios.  I  further  described  the  process  by 
conducting  a  practical  example  with  each  participant.  Using  a  website,  I  described  my 
actions  while  navigating  the  site.  For  example,  I  might  state,  “I’m  looking  for 
information  on  shopping  so  I  think  I  will  click  on  this  shopping  button. ..Oh,  that  didn’t 
work  like  I  thought.  OK  I’ll  try....”  Participants  were  then  asked  to  confirm  their 
understanding  of  thinking  aloud. 

While  participants  conducted  tasks,  I  did  not  intervene  in  any  way.  Participants 
were,  however,  reminded  to  continue  thinking  aloud  when  they  discontinued  doing  so.  I 
employed  a  neutral,  standard  prompt,  asking  participants,  “Can  you  tell  me  what  you  are 
doing  now?”  This  method  was  employed  in  order  to  solicit  continued  verbalization 
without  interjecting  bias. 

The  think  aloud  sessions  were  video  recorded  and  I  later  transcribed  observations 
and  participant  statements  for  trend  analysis.  Where  trends  were  found,  selected  verbal 
data  from  the  think  aloud  process  were  combined  with  post-completion  interview  data  for 
incorporation  into  the  analysis  and  findings. 

3.  Qualitative,  Semi-structured  Interview 

Upon  completion  of  the  scenarios,  participants  were  asked  to  discuss  what  they 
had  experienced  while  working  with  the  system.  I  maintained  a  neutral  approach  to 
interviewing  in  order  to  reduce  potential  bias.  I  accomplished  the  neutrality  by  refraining 
from  leading  questions  which  may  have  driven  participants  toward  a  particular  response. 

The  interview  period  began  with  the  question,  “Can  you  talk  about  your 

experience  with  the  system?”  Follow  up  questions  were  asked  in  reference  to 

participants’  specific  actions  while  completing  the  task.  These  follow  up  questions 

focused  on  suspected  problem  areas  the  participant  had  experienced  during  the  test. 

However,  these  questions  were  presented  in  an  open-ended  syntax,  also  to  avoid  bias  and 

solicit  richer  responses.  For  example,  I  would  state,  “While  you  were  reading,  you 

commented,  ‘I  need  to  go  back  up  a  little,’  could  you  discuss  that  a  bit?”  Furthennore,  I 
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avoided  using  affirmations  that  might  suggest  agreement  or  otherwise  influence  a 
participants’  thought  process.  The  interviewer  typically  acknowledged  statements  by 
using  non-evaluative  remarks  such  as,  “OK,”  or  “mmmhmm,”  and  then  continued  to  the 
next  area  of  interest. 
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III.  ANALYSIS  OF  RESULTS 


In  this  chapter,  I  discuss  the  findings  from  my  analysis  of  trends  in  the  user  data. 
The  findings  relate  to  three  areas:  using  the  system  in  general,  locating  documents,  and 
using  documents  themselves.  Verbal  analysis  was  conducted  in  three  stages.  First,  the 
video  recordings  were  reviewed  to  get  an  overall  sense  of  participant  actions  and 
commentary.  Second,  the  videos  were  reviewed  again  and  activity  and  commentary  were 
transcribed.  Third,  the  transcriptions  were  analyzed  to  isolate  potential  themes.  Last,  the 
transcriptions  were  reviewed  again  to  refine  the  data  and  ensure  that  the  emergent  themes 
held  constant. 

Overall,  I  found  that  users  felt  the  system  is  adequate  and  supports  their  work,  yet 
search  limitations  as  well  as  incomplete  and  inconsistent  document  libraries  erode  their 
confidence  in  using  it.  The  data  also  show  that  users  were  not  able  to  complete  all  tasks 
within  the  time  limit  by  using  this  system.  Further,  it  was  evident  that  search  and 
navigation  deficiencies  were  a  contributing  factor. 

Additionally,  I  found  that  users  are  seeking  some  capabilities  that  do  not  exist  in 
the  current  system.  Specifically,  users  desire  a  capability  to  make  documents 
personalized  and  portable  so  that  they  can  easily  access  documents  they  use  repeatedly, 
and  users  want  to  make  notes  or  annotate  areas  within  documents  that  they  commonly 
use.  Each  finding  is  presented  in  a  detailed  data/discussion  format  which  follows. 

A.  USERS  RANKED  THE  EXPERIENCE  AS  MODERATELY  EASY 

The  data  show  that  users  believe  that  the  system  supports  their  work  and  is  not 
prohibitively  difficult  to  use.  However,  usability  rankings,  user  comments,  and 
observation  of  problem  areas  indicate  that  users  had  a  moderate  level  of  difficulty  overall. 
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1. 


User  Data 


My  survey  data  indicate  that  users  found  the  system  moderately  easy  to  use: 


•  Participants  ranked  the  overall  usability  (SUS)  71.5%  on  a  0%-100% 
scale 

•  “I  found  the  system  supports  my  work.”  (Mean  response  of  4.0  on  a  5 
point  scale) 


Users  commented  positively  that  the  Marine  Corps  website  (www.usmc.mil) 
assists  their  efforts  in  locating  information: 


•  “I  knew  where  to  go... .I’ve  used  that  [www.usmc.mil]  before... If  you 
click  on  publications  it  brings  you  to  everything,  forms,  orders, 
directives. .  .That’s  not  hard  to  navigate” 

•  “I  know  immediately  to  go  to  the  Marine  Corps  website.... It  breaks  it 
down  pretty  clearly  how  you  want  to  look  them  up... If  I’m  able  to  do 
keyword  searches  it  saves  a  lot  of  time” 


Users  expressed  some  difficulty  with  locating  information  on  the  site: 


•  “It’s  all  right  there  if  you  can  find  it....  I  usually  use  Google... once  I 
figure  out  what  I’m  looking  for  then  I  can  go  to  [www.Jusmc.mil  because 
it  has  the  information  out  there” 

•  “I  didn’t  feel  it  was  bad.... Once  I  found  what  I  was  looking  for . I  had 

to  search  around  a  little  bit. . .  .1  didn’t  think  it  was  bad” 

•  “. .  .Not  sure  where  it  is  on  here. .  .I’m  sure  it’s  on  here  somewhere.” 
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And  users  often  left  the  site  to  conduct  keyword  searches: 


•  5  of  5  participants  abandoned  the  site  to  use  commercially  available  tools 

(Google)  when  searching  for  keywords  (2  after  failed  attempts  to  first  use 
the  keyword  search  on  the  site) 

2.  Discussion 

Results  from  the  SUS  questionnaire  and  users’  general  reaction  to  using  the 
system  were  relatively  positive.  The  SUS  score  (71.5%)  and  the  subsequent  user 
comments  suggest  that  users  are  tolerant  and  accepting  of  the  current  system  which,  as 
indicated  by  the  positive  assessment,  does  help  support  their  work  (mean  score  4.0).  In 
large  part,  this  positive  attitude  seems  to  be  centered  on  the  idea  that  users  felt  confident 
that  the  documents  could  be  found  from  the  official  USMC  website,  citing,  “/  knew 
where  to  go,  ”  and  “it  has  the  information  out  there.'’’’ 

Some  aspects  of  the  user  comments  may  indicate  that  while  the  overall  experience 
was  positive,  elements  affecting  its  ease  of  use  may  need  improvement.  For  example, 
when  interviewed,  users  inserted  qualifiers  such  as  “it’s  all  right  there  if  you  can  find  it ,” 
“I  had  to  search  around  a  little  bit."  and  “I’m  sure  it’s  on  here  somewhere.  ”  Yet  another 
user  pointed  out  the  site’s  limited  ability  to  help  detennine  which  documents  were  needed 
to  accomplish  a  task,  “I  usually  use  Google.  Once  I  figure  out  what  I’m  looking  for,  then 
I  can  go  to  usmc.mil.” 

Another  indication  of  user  difficulty  was  evident  because  users  often  abandoned 
looking  for  documents  from  the  USMC  site  after  encountering  unexpected  or  blank 
search  results  or  anytime  a  keyword  search  needed  to  be  conducted.  The  use  of  qualified 
statements  and  abandonment  of  the  site  indicates  that  while  users  expressed  overall 
optimism  about  the  system,  some  elements  of  effectiveness  and  efficiency  reduce  the 
users’  overall  satisfaction  with  using  it. 
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B.  USERS  SEEK  REASSURANCE  THAT  INFORMATION  IS  CURRENT 

AND  COMPLETE 

The  survey  results  and  user  comments  also  indicated  that  users  may  lack 
confidence  that  the  system  will  provide  complete  and  current  information,  although  some 
positive  data  exist  in  this  area  as  well.  Users  tended  to  indicate  that  they  thought  the 
information  would  be  current,  but  expressed  some  concern  as  to  whether  it  actually  was, 
in  fact,  current. 

1.  User  Data 

Collectively,  users  responded  neutrally  in  the  post-test  survey  when  asked  to 
report  on  their  confidence  in  finding  all  of  the  salient  information  by  using  the  system: 


•  “I  was  fairly  confident  that  I  found  all  the  infonnation  by  using  this 
system.”  (Mean  response  of  3.0  on  a  5  point  scale) 


User  comments  further  identified  a  lack  of  confidence  that  simply  retrieving 
information  online  will  provide  complete  and  reliable  answers  to  their  work-related 
questions: 

•  “I  choose  the  paper  version  because  there  is  a  choice . I  think  that  bleeds 

over  into  the  confidence  in  the  documents.” 

•  “. .  .1  would  probably  take  more  time  to  be  sure  of  that  [infonnation  found] 
and  bounce  it  off  what  I  find  in  other  documents  before  I  confirm 
anything.” 

•  “I  would  have  gone  to  the  chief  to  make  sure...  [What  are  you  doing 
now?]  Going  back  through  to  double  check. . .  If  I’m  going  to  talk  like  an 
expert  I  wanted  to  reread  it” 
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Post-test  survey  data  were  more  positive  in  regard  to  the  currency  of  information, 
but  still  indicates  mild  reservations: 


•  “I  was  fairly  confident  that  the  information  was  the  most  current 
available.”  (Mean  response  of  3.6  on  a  5  point  scale) 


User  comments  generally  supported  that  they  thought  information  would  be 
current  due  to  the  fact  that  it  is  electronically  updated  and  maintained,  but  they  were  not 
assured  of  this  currency,  which  they  felt  to  be  important: 


•  “Maybe  it’s  just  blind  faith,  but  it  seems  like  it’s  updated  pretty 
regularly.... the  USMC  site... I  don’t  know  if  it’s  just  trust.... I  never  felt 
like  I  would  get  the  wrong  thing. .  ..I  have  confidence.” 

•  “You’d  want  to  make  sure  it’s  up  to  date .  this  type  of  infonnation 

changes  on  a  yearly  basis.... I  feel  that  having  the  most  up  to  date  is 
important. . .  .electronic  documents  are  going  to  be  the  most  up  to  date.” 

•  “I  didn’t  check  the  date  but  I  have  to  assume  through  the  USMC  site 
they’ll  publish  the  most  up  to  date  stuff.” 

•  “...the  last  time  I  was  doing  supply  stuff,  it  wasn’t  really  updated.  I 
thought  about  going  here  [www.usmc.mil].... but  didn’t  because  it  wasn’t 
a  real  positive  experience.” 

•  “I  don’t  know  how  old  this  document  is...  Maybe  it’s  updated  more  often 
than  this...  [referring  to  RCO  site]... I  would  assume  these  guys  are  the 
head. .  .they’ve  got  the  [most  current  info].” 

•  “This  document  is  not  up  to  date. .  .it’s  talking  about  [limit  of  $2500]  when 
I  know  its  up  to  $3000. .  .But  I’ll  use  it  nonetheless. . .  that  would  make  me 
a  little  nervous  that  information  I  take  out  of  here  may  not  be  entirely 

accurate  ....  I  would  follow  up  with  a  call  to  the  contracting  office.” 
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•  “I  feel  this  is  current  and  up-to-date. . .  [www.Jusmc.mil  everything  is  up- 
to-date.  The  latest  and  greatest....  I  feel  more  confident  on  here  than  what 
I  would  find  in  a  pub  binder  [printed].” 

2.  Discussion 

As  participant  four  accurately  points  out,  trust  is  a  key  factor  in  using  infonnation. 
Policy  providers  therefore  need  to  ensure  that  their  information  supports  “ confidence  in 
the  documents’'’  and  “ having  the  most  up-to-date ”  information.  Yet  users  found 
information  which  ‘ may  not  be  entirely  accurate ,’  ‘maybe ...updated  more  than  this,'  and 
were  uncertain  as  to  ‘ how  old  this  document  is .’  These  comments  indicate  some  concern 
exists  over  currency  of  information,  particularly  during  scenario  two,  when  users 
retrieved  information  from  locations  other  than  www.usmc.mil.  Overall,  however,  users 
indicated  that  they  felt  www.usmc.mil  provided  ‘ the  latest  and  greatest  and  they 
expressed  confidence  through  a  marginally  positive  survey  response  (3.6).  This  marginal 
confidence  is  apparently  representative  of  user  reliance  on  ‘ blind  faith,'  and  “ assuming ” 
that  the  sites  are  up-to-date,  rather  than  an  explicit  understanding  that  the  information  is 
current. 

Another  aspect  of  trust  is  the  completeness  of  the  information.  Users  indicated  in 
the  survey  that  they  somewhat  lacked  confidence  that  all  infonnation  had  been  located 
(mean  response  3.0).  This  shortcoming  is  evidenced  in  user  comments  indicating  they 
would  ‘choose  paper,’  ‘bounce  off  other  documents,’  or  consult  other  sources  to  ensure 
reliability  and  completeness  of  information. 

C.  USERS  WERE  UNSUCCESSFUL  IN  LOCATING  POLICY  DOCUMENTS 

FOR  A  MARINE-CORPS-  SPECIFIC  TASK 

Although  I  have  noted  that  users  indicated  knowledge  of  the  www.usmc.mil  web 
site  as  a  primary  source,  users  experienced  difficulty  using  the  site  to  locate  the  document 
required  to  complete  a  Marine-Corps-specific  task.  This  was  evidently  the  result  of 
inadequate  search  functionality  and  a  mismatch  between  the  site  layout  and  the  way  users 
view  the  content.  Both  items  will  be  discussed  further  in  later  sections. 
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1. 


User  Data 


Users  had  difficulty  locating  documents  within  the  USMC  web  site: 


•  4  of  5  participants  failed  to  locate  the  policy  document  required  to 
complete  the  property  control  scenario  within  the  time  limit. 

Most  users  did  not  use  the  site  search  function,  and  those  that  did  were 
unsuccessful: 

•  5  of  5  participants  used  www.usmc.mil  in  combination  with  Google  as 
their  primary  search  mechanism 

•  2  of  2  participants  failed  when  using  the  www.usmc.mil  search  box  as  a 
primary  method  for  searching 

•  3  of  5  participants  did  not  attempt  to  use  the  www.usmc.mil  search  box 


Only  one  user  was  able  to  find  a  document  through  site  navigation: 


•  2  of  3  participants  failed  when  using  general  site  navigation  as  a  primary 
method  augmented  by  Google  keyword  searches 

•  1  participant  succeeded  by  using  navigation,  choosing  the  option  to 
browse  ‘all  MCOs  by  SSIC’ 

2.  Discussion 

Users  attempted  to  retrieve  documents  by  navigating  within  the  site  (3  users),  by 
using  the  site’s  internal  search  capability  (2  users),  and  by  augmenting  their  search  with 
Google.com  (5  users).  All  three  methods  proved  to  be  ineffective,  resulting  in  the  high 
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failure  rate  (4  of  5  users).  The  one  user  that  was  successful  navigated  to  the  document 
and  located  it  by  its  document  number.  The  other  four  users,  who  did  not  know  or  search 
by  document  number,  were  unsuccessful.  This  indicates  two  things:  first,  that  document 
number  knowledge  is  useful  for  searching,  but  second,  most  users  do  not  search  in  this 
way.  The  other  four  users  searched  instead  using  key  tenns  drawn  from  the  task,  such  as 
“CMR,”  “SL-3”  and  “accountability.”  In  subsequent  sections,  the  discussion  of  specific 
search  and  navigation  difficulties  provides  additional  observations,  which  may  clarify  the 
users’  view  of  the  task  and  problem  space. 

D.  USERS  WERE  MODERATELY  SUCCESSFUL  IN  LOCATING  POLICY 

DOCUMENTS  FOR  A  NON-MARINE-CORPS-SPECIFIC  TASK 

Users  had  greater  success  with  a  wider  topic  area  by  employing  broad  internet 
search  capability  via  Google.  All  users  were  guided  by  the  awareness  of  a  potential 
source  for  the  infonnation  (Regional  Contracting  Office  (RCO)).  However,  users  did  not 
seem  to  know  how  to  get  to  the  source  directly.  Only  one  user  relied  on  the 
www.usmc.mil  site,  and  that  user  was  unsuccessful. 

1.  User  Data 

Users  experienced  greater  success  in  locating  a  document  that  was  not  specific  to 
the  Marine  Corps: 


•  3  of  5  participants  succeeded  in  locating  a  policy  document  required  to 

complete  the  financial  management  scenario  within  the  time  limit 
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The  increased  success  was  not  attributable  to  www.usmc.mil.  In  fact,  users  either 


didn’t  consider  it  as  an  option  or  failed  by  using  it: 


•  2  participants  never  considered  the  site 

•  “I  never  think  to  look  on  USMC  site  for  credit  card  info. . ..I  didn’t 
even  look  on  USMC  site... it  could  very  well  be  there”  (participant 
succeeded  using  Google) 

•  “Is  it  the  FAR?. .  .I’ll  try  DoD. .  .Which  agency  would  be  in  charge 
of  that. .  .I’ll  go  to  DAU. . .”  (participant  failed  using  DAU) 

•  1  participant  failed  by  attempting  to  navigate  to  the  RCO  site  through 
www.usmc.mil) 


Successful  users  primarily  used  broad  internet  search  capability  (Google)  and  user 
knowledge  of  a  potentially  viable  source  (Regional  Contracting  Office  (RCO)).  One  user 
found  the  site  immediately: 


•  1  used  Google  to  search  for  the  RCO  and  then  navigated  to  a  card  holder 

(CH)  desk  guide  on  the  site 

However,  two  other  users  were  either  ‘lucky’  or  used  a  back-door  method  to  find 
a  source: 


•  1  participant  used  Google  to  search  for  ‘regional  contracting  southwest,’ 

then  opened  a  PowerPoint  file  returned  by  the  search,  then  identified  the 
RCO  URL  embedded  in  the  Power  Point,  and  only  then  retrieved  the 
Approving  Official  (AO)  desk  guide. 

•  The  participant  commented,  “The  USMC  publications... it’s  just 
hard  to  search  if  you  don’t  know  the  exact  title  or  number,  which  I 
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will  usually  try  to  find  it  through  Google  or  my  paper 
documents...” 

•  1  participant  used  Google  to  search  for  ‘government  commercial  purchase 

card,’  then  followed  a  link  to  the  HQMC  GCPC  SOP. 

•  The  participant  commented,  “I  could  have  searched  all  day  and  not 

found  it . I  just  got  lucky  that  exactly  what  I  want  popped  up. .  ..I 

could  have  gone  back  to  USMC....I  don’t  know  exactly  where  it  is 
on  there....  I  guess  I  got  lucky  really....  I  just  did  a  Google 
search... and  I  got  lucky  a  current  SOP  came  up  when  I  clicked  on 
the  link” 

2.  Discussion 

Interestingly,  only  one  user  attempted  to  retrieve  a  policy  document  for  this 
scenario  by  using  the  www.usmc.mil  website  and  that  user  was  unsuccessful.  Of  the 
three  users  that  were  successful  in  attaining  a  valid  document,  none  used  the  USMC  site. 
One  used  the  HQMC  GCPC  SOP  retrieved  via  Google,  and  two  used  GCPC  desk  guides 
retrieved  through  the  RCO  web  site,  also  located  via  Google.  These  methods,  combined 
with  user  comments,  indicate  that  while  the  policy  information  is  available  from  within 
the  Marine  Corps,  users  are  not  confident  on  where  to  obtain  it:,  “I  could  have  gone  back 
to  USMC... I  don  7  know  exactly  where  it  is  on  there... I  guess  I  got  lucky,  ”  or  “/  never 
think  to  look  on  the  USMC  site  for  credit  card  info,  ”  and  “it’s  just  hard  to  search  if  you 
don  7  know  the  exact  title  or  number.  ”  Users,  then,  resort  to  internet  searching,  despite 
the  designation  of  www.usmc.mil  as  the  official  site  for  policy  within  the  Marine  Corps. 
One  user  even  resorted  to  a  back-door  method,  locating  the  website  from  a  PowerPoint 
file  loaded  on  a  server.  While  users  should  have  options  available  to  them  and  may  still 
choose  to  search  for  policy  in  familiar  ways  (like  Google),  the  utilization  of 
www.usmc.mil  may  prove  more  effective  if  all  available  policy  guidance  were  current 
and  available  there. 
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E.  LACK  OF  A  SINGLE,  REPUTABLE,  CONSOLIDATED  SOURCE 

CONTRIBUTED  TO  INEFFICIENT  SEARCH  AND  NAVIGATION 

Users,  on  average,  used  between  three  and  six  pathways  in  their  attempts  to  locate 
documents.  Users  that  did  not  know  the  source  document  by  title  or  numerical  identifier 
had  difficulty  determining  the  location  of  the  document.  Users  commented  that  a  ‘one 
stop  shop’  would  be  a  preferred  addition  to  the  current  organization  to  help  alleviate  this 
problem. 

1.  User  Data 

Users  commonly  resorted  to  inefficient  methods  of  locating  documents: 


•  An  average  of  6  pathways  were  used  to  locate  documents  for  scenario  one: 

•  Units  =>  by  location  =>  North  Carolina  =>  MCCSSS; 

•  Publications=>Marine  Corps  Publications=>PLMS; 

•  Publications=>Marine  Corps  Publications=>Technical  Manuals; 

•  Publications=>Orders/Directives=>Misc  Pubs; 

•  Searchbox  using ‘Orders’ filter; 

•  Searchbox  using  ‘Directives’  filter 

•  Publications  (from  the  drop  menu  -  clicking  directly  as  if  it  were  a 
link,  when  it  is  not) 


•  An  average  of  3  pathways  were  used  to  locate  documents  for  scenario  two: 

•  Google=>DAU  web  site: 

•  Units=>Alphabetically=>Camp  Pendleton  (looking  for  contracting 
office); 
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•  Publications  (from  the  drop  menu  -  clicking  directly  as  if  it  were  a 
link,  when  it  is  not) 

•  Google=>(in  search  of  RCO  web  site) 


Only  one  user  knew  the  document  number  and  was  successful  in  navigating  the 
USMC  web  site  to  find  the  document,  other  users  were  unsuccessful  on  the  site: 

•  1  of  6  navigational  paths  followed  on  www.usmc.mil  resulted  in  the 
successful  location  of  a  document  [browse  by  SSIC  (participant  knew  the 
document  number)]. 

•  The  successful  participant  commented,  “I  think  it’s  actually 
organized  really  well  right  now.... you  do  have  the  options  and  it’s 
all  very  clear,  straightforward.” 

•  5  of  6  navigational  paths  followed  on  www.usmc.mil  were  unsuccessful. 
Paths  included  Publication  Library  Management  System  (PLMS), 
miscellaneous  pubs,  Technical  Manuals,  and  others. 

Users  further  commented  on  the  need  to  know  document  identifiers: 

•  “I’m  looking  for  the  order/pub,  for  the  one  that  lists  the  TAMs....I  don’t 
know  what  that  is... [checks  paper  document,  goes  to  Google]... the  other 
thing  is,  just  call  division.” 

•  “I’m  going  on  a  wild  goose  chase  I  have  no  idea... Just  googling  random 
[things]  to  see  what  I  can  find. . .  I  don’t  know  the  name  of  the  pubs.” 
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Users  also  expressed  that  a  consolidated  listing  of  all  documents  would  help  meet 
their  needs: 

•  “I  prefer  a  reputable  one-stop  shop... maintained  with  semblance  of 
authority” 

•  [If  you  could  organize  how  you  wanted?]...  “Listed  by  the  SSIC  code52 
regardless  of  whether  it’s  a  user  manual,  or  P-directive53  or 
category. .  ..listing  all  by  code  it  might  be  easier  like  if  you  didn’t  know 

exactly  what  you  were  looking  for . not  to  get  rid  of  everything  else  but 

just  another  option . so  it’s  geared  toward  each  person’s  preference” 

•  “I  thought  there  was  a  way  you  could  just  list  them  all  here,”  [and  later, 
after  two  failed  site  searches  and  attempting  to  use  the  ‘Publications’ 
menu  option] ...  ’Is  this  the  way  you  get  the  entire  listing?' 

•  “....I  think  my  only  problem  is  not  all  the  pubs  are  on  there 
[www.usmc.mil]... What  I  really  want  is  the  [user  manual#]  124  and  I 
don’t  know  where  to  get  it. .  .I’ve  looked  for  some  pubs  that  were  not  listed 
there  [www.usmc.mil]  and  I’m  not  sure  why.” 

2.  Discussion 

In  looking  for  policy  documents,  users  did  not  seem  to  have  an  efficient  means 
through  which  they  could  locate  the  document  desired,  when  looking  for  Marine  Corps 
specific  documents  (average  of  6  methods  used).  The  only  successful  navigation  method 
on  the  site  was  to  seek  a  document  by  its  numerical  identifier  (1  of  6  site  navigation 
attempts),  yet  most  users  did  not  know  the  document  identifier.  For  example,  “I’m 

52  SSIC  is  the  Standard  Subject  Identification  Code,  a  taxonomy  used  throughout  Naval 
documentation  to  organize  documents  by  operational  category.  SSIC  is  itself  controlled  by  regulation.  See 
Department  of  the  Navy  SECNAV  M-5210.2,  Standard  Subject  Identification  Code  (SSIC)  Manual 
(December  2005). 

53  Under  the  current  organization,  P-directives  are  listed  separately  and  do  not  fall  in  numerical 
sequence  with  other  directives  and  documents  which  are  not  preceded  by  a  ‘P’  in  the  document  number. 
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looking  for  the  order/pub,  for  the  one  that  lists  the  TAMs..../  don ’t  know  what  that  is” 
and,  “I’m  going  on  a  wild  goose  chase.../  don’t  know  the  name  of  the  pubs.'”  Thus, 
document  identifiers  alone  do  not  support  the  manner  in  which  users  look  for 
information. 

Users  sought  documents  in  varying  ways  based  on  the  information  from  the 
scenario  or  other  key  identifying  infonnation  relative  to  the  task.  However,  users  did  not 
always  have  a  clear  concept  of  where  that  information  might  be  located:  “I  thought  there 
was  a  way  you  could  just  list  them  all  here.  ”  And  “I’ve  looked  for  some  pubs  that  aren ’t 
listed  there,  and  I’m  not  sure  why”  Users  also  requested  a  complete,  consolidated  view 
of  the  information  that  is  available,  stating,  “I  prefer  a  one  stop  shop  from  a  reputable 
source.  ”  These  findings  are  consistent  with  usability  best  practices  which  suggest  that 
sites  should  maintain  multiple  methods  of  entry  to  information  and  be  consistent  with  the 
ways  people  do  their  work.54 

F.  USERS  LACKED  CONFIDENCE  THAT  THE  USMC  WEBSITE 
1WWW.USMC.MIL)  SEARCH  ENGINE  WOULD  PROVIDE  ADEQUATE 
RESULTS 

Users  verbally  stated  their  lack  of  confidence  in  the  USMC  search  tool.  Users 
that  attempted  to  use  the  search  tool  found  no  relevant  results.  As  a  result,  most  users 
frequently  relied  on  Google  to  conduct  searches  when  looking  for  policy  documents, 
despite  commonly  acknowledging  that  the  www.usmc.mil  site  is  the  primary  resource  for 
policy  documents. 


54  U.S.  Department  of  Health  and  Human  Services,  Research-Based  Web  Design  &  Usability 
Guidelines,  Chapter  two.  http://www.usability.gOv/pdfs/guidelines.html#2  (accessed  December  1,  2007). 
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1. 


User  Data 


Users  generally  expressed  a  lack  of  confidence  that  the  search  box  on 
www.usmc.mil  would  be  helpful: 


•  “You  have  to  know  exactly  what  you’re  looking  for.... It’s  hard  to 
search.... the  search  engine  on  the  website... I  haven’t  had  much  success 
with. .  .1  usually  will  go  to  Google  or  look  in  my  [paper-based]  pubs.” 

•  “You  generally  don’t  use  [the  www.usmc.mil  search  tool]  because  it’s 
never  really  done  what  I  wanted... To  me,  as  a  user,  it  seems  like  the 
search  engine  is  kind  of  weak.” 

•  “The  times  I  have  used  the  search  box  hasn’t  brought  up  what  I 
want. . .  .it’s  not  the  most  user  friendly. . .” 

•  [What  would  make  your  experience  better?]  “Maybe  a  Google -type  search 

engine  on  usmc.mil  where  it  brings  up  kind  of  what  I’m  looking  for . a 

stronger,  more  user  friendly  search  engine  on  the  USMC  website. . .” 


This  lack  of  confidence  led  users  to  rely  primarily  on  Google  for  searching: 


•  5  of  5  participants  eventually  abandoned  the  www.usmc.mil  site  for 
Google  when  attempting  to  search  by  keyword 

•  2  of  5  participants  attempted  the  www.usmc.mil  search  box  first,  and  then 
immediately  abandoned  the  site  for  Google  after  getting  zero  results 
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Users  that  attempted  to  use  the  search  capability  were  confused  by  the  search 
filters,  and  abandoned  it  due  to  lack  of  search  results: 


•  1  participant  tried  to  search  without  a  keyword  [seeking  a  full  list  of 
publications],  and  then  with  the  keyword  ‘supply  manual’  using  the  orders 
filter  and  then  again  by  using  the  directives  filter  -  all  three  returned  zero 
results 

•  The  participant  commented  [after  zero  search  results],  “I’d  stick 
with  Google  at  this  point. .  .I’m  getting  something” 

•  And  later,  during  the  interview,  “I  was  looking  for  orders  and  the 
dropdown  menu  reset  to  directives...  until  you  get  results  it’s  not 
clear  . .  .did  it  search  for  orders  or  did  it  search  for  directives?” 

•  1  participant  attempted  to  search  for  ‘accountability,’  first  using  the  orders 
filter,  then  again  with  the  directives  filter.  Both  returned  zero  results. .  .the 
user  abandoned  the  site  for  Google 

2.  Discussion 

Users  avoided  using  the  search  tool  on  the  www.usmc.mil  website,  indicating  that 
they  “ haven’t  had  much  success  with  [it],  ”  “it’s  never  really  done  what  I  wanted,  ”  or 
“it’s  not  the  most  user-friendly.  ”  All  of  the  users  in  the  study  primarily  used  Google  to 
search,  presumably  in  part  due  to  this  impression  of  the  site’s  ‘ kind  of  weak’  search 
capability. 

The  search  capability  of  the  website  should  be  easy 
to  use  and  provide  expected  results.  In  this  study,  the 
search  tool  was  deficient  in  both  areas.  First,  users  did  not 
find  the  tool  easy  to  use.  The  two  users  that  chose  to  use 
the  tool  both  had  difficulty  with  the  search  filter,  first 
applying  the  ‘orders’  filter,  and  then,  after  no  results, 
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applying  the  ‘directives’  filter,  which  also  returned  zero  results  for  their  search  terms  (See 
Figure  2).  One  user  noted  that  regardless  of  the  filter  applied,  the  tool  defaulted  back  to 
‘directives’  after  the  search  was  complete  and  he  commented,  “It’s  not  clear. ..did  it 
search  for  orders  or  did  it  search  for  directives?”  Additionally,  since  both  users 
attempted  the  search  for  their  keyword  using  each  filter  successively,  it  would  appear  that 
users  do  not  understand  what  areas  of  the  site  correspond  to  the  different  search  filter 
selections  ‘orders’  and  ‘directives.’  detracting  from  the  search  tool’s  ease  of  use.  These 
data  suggest  that  the  search  filter  may  need  improvement  to  alleviate  user  confusion. 

Second,  the  search  tool  did  not  provide  expected  results.  The  users  conducted  the 
searches  using  terms  they  suspected  to  be  in  the  target  document  title  or  text.  On  each 
occasion  that  the  search  tool  was  used,  zero  results  were  returned  (both  users,  either 
filter).  The  search  term  ‘‘accountability,'  for  example,  is  prevalent  throughout  the  target 
document,  yet  the  search  engine  did  not  return  any  results  for  this  search  term.  The  term 
‘ supply  manual’  should  have  returned  the  intended  source  document  (from  the  title  - 
Marine  Corps  Consumer  Level  Supply  Policy  Manual )  yet  it  did  not.  These  examples 
highlight  that  the  search  engine  does  not  perform  the  way  users  expect  and  causes  users 
to  leave  the  site.  As  one  user  indicated,  “ I’d  stick  with  Google  at  this  point.  I’m  getting 
something.  ” 

G.  USERS  NEED  BETTER  WAYS  TO  ACCESS  AND  EASILY  RETURN  TO 

THEIR  FREQUENTLY  USED  DOCUMENTS 

Some  users  indicated  the  repetitive  nature  of  their  research  and  expressed  a  desire 
to  maintain  availability  of  their  commonly  used  documents  without  searching  again  or 
downloading  them  each  time. 

1.  User  Data 

Users  repeatedly  use  the  same  source  documents  for  information: 


•  “There’s  like  two  pubs  used  for  99%  of  stuff  we  did...  for  the  majority  of 
answers  you  go  through. .  .you  go  to  the  same  stuff  over  and  over  again.” 
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Additionally,  users  seek  ways  to  globally  access  those  documents: 


•  “I’ll  use  Google  or  USMC  [www.usmc.mil]  because  I  may  not  be  at  my 
computer. .  .1  won’t  have  my  document  with  me. . . .  Maybe  [if  it  brought  up 
my  desktop. . .]  that  way  I  can  reference  it.” 


Without  this  feature  available,  users  attempt  to  create  portability  in  their  own 

way: 


•  “I  usually  use  my  own  stuff. . .”  [referring  to  paper  documents] 

•  “Normally,  a  lot  of  this  stuff  is  bookmarked” 


The  nature  of  the  military  environment  also  reinforces  the  need  for  global  access: 


•  “When  I  was  at  my  old  unit,  I  actually  had  that  downloaded  to  my  desktop 
[while  observer  retrieved  the  primary  source  doc]...  I  didn’t  want  to  take 
time  to  download  everything.” 

2.  Discussion 

In  addition  to  the  ‘one-stop  shop’  previously  identified,  users  are  concerned  about 
document  organization  and  availability  as  it  relates  to  their  common  needs.  As  one  user 
commented,  “You  go  to  the  same  stuff  over  and  over  again.”  Naturally,  then,  users  do 
not  want  to  repeatedly  search  for  the  same  documents.  These  data  imply  that  efficiency 
may  be  increased  by  giving  users  an  option  to  customize  a  policy  screen,  which  would 
allow  them  to  electronically  create  the  personalized,  portable  access  desired  and  “use 
their  own  stuff’.  Another  option  might  be  a  ‘frequently  used’  component  for  the 
interface  where  people  can  quickly  see  the  most  frequent  items  used. 
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H.  USERS  WERE  NOT  SUCCESSFUL  IN  QUICKLY  LOCATING  ALL  THE 

INFORMATION  NEEDED  FOR  DECISION-MAKING 

The  data  show  that  users  who  referred  to  procedural  documents  had  more  success 
in  locating  information  to  answer  questions  than  those  that  strictly  used  policy 
documents.  Those  users  who  were  seeking  procedural  information  in  policy  documents, 
(a  common  occurrence  as  we  have  shown)  were  unsuccessful  in  locating  answers. 

1.  User  Data 

In  scenario  one,  most  users  found  partial  answers  but  were  not  able  to  answer  the 
Commander’s  question  regarding  how  to  properly  account  for  UURI  items: 

•  4  of  5  participants  correctly  identified  the  Commanding  Officer  as  the 
proper  authority  to  establish  allowances 

•  5  of  5  participants  consulted  MCO  P4400. 150E  (either  located  or  provided 
to  them)  as  the  source  document  for  UURI 

•  0  of  5  participants  located  the  information  on  how  to  account  for  UURI 
component  items  within  this  policy  document 


In  scenario  two  some  users  found  enough  information  to  advise  the  Commander 
against  the  purchase  by  using  guides  and  SOPs,  but  none  found  all  the  information 
requested: 

•  0  of  5  participants  located  all  information  requested 

•  3  of  5  participants  determined  that  the  purchase  was  unauthorized  by  using 
desk  guides  or  the  HQMC  SOP  (procedural  documents)  as  a  source 
document 

•  1  of  the  3  participants  commented,  “I  go  to  CH  desk  guide  because 

it  has  ‘in  the  weeds’ . . .  .answers” 
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Users  provided  the  policy  document  NAVSUPINST  4200.99,  however,  were 
unsuccessful  in  making  a  determination: 


•  2  of  5  participants  were  provided  NAVSUPINST  4200.99  as  a  source 

document 

•  1  of  the  2  participant  prematurely  advised  that  their  was  ‘a 
possibility’  that  the  purchase  could  be  authorized  under  the 
necessary  expense  rule  (NAVSUPINST  4200.99  as  source) 

•  1  of  the  2  participants  made  no  advisement  (time  expired) 

2.  Discussion 

Users  who  referred  to  policy  documents  (MCO  4400. 150E  for  scenario  one  and 
NAVSUPINST  4200.99  for  scenario  two  were  unsuccessful  in  finding  all  required 
information  to  complete  the  task.  Users  who  referred  to  procedural  documents  like  desk 
guides  (2  users)  or  SOPs  (1  user)  were  able  to  find  the  information  that  enabled  them  to 
accurately  advise  the  Commander  in  scenario  two.  Yet,  the  tasks  required  users  to  cite 
policy  on  the  subject,  not  to  describe  the  procedures  they  would  use.  This  indicates  that 
policy  and  procedural  documents  may  not  be  entirely  severable,  suggesting  a  need  for  a 
more  comprehensive,  integrated  system  of  information  availability.  In  fact,  users  seemed 
to  prefer  to  use  procedural  documents  to  find  policy  answers  (3  of  5  users).  As  one 
successful  user  indicated,  “/go  to  the  desk  guide  because  it  has  ‘in  the  weeds  ’  answers.  ” 

As  previously  evidenced  from  the  survey  results  and  validation  of  the  scenarios, 
the  user  group  is  typically  driven  to  policy  documents  to  solve  procedural  questions  to 
ensure  that  procedures  are  in  accordance  with  policy.  Since  policy  documents  are  not 
necessarily  organized  by  procedure,  users  may  be  more  inclined,  and  more  successful, 
when  using  procedural  documents  in  place  of  the  source  policy  document  because  they 
reflect  the  user’s  goals  more  directly. 
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I.  USERS  HAD  DIFFICULTY  MOVING  WITHIN  DOCUMENTS 

Users  spent  a  lot  of  time  scrolling  within  documents  to  locate  infonnation  referred 
to  from  other  references  within  the  document  (for  example,  the  Table  of  Contents  (TOC) 
and  chapter  headings).  To  circumvent  the  problem,  users  initiated  work-arounds,  but 
suggested  enhancing  documents  with  better  search  capability  and  internal  hyperlinking. 

1.  User  Data 

Users  unsuccessfully  attempted  to  locate  infonnation  within  documents  using 
keyword  searches. 

•  3  of  5  participants  attempted  to  search  for  a  keyword,  but  moved  to  the 

TOC  after  receiving  ineffective  results 


Users  scrolled  between  the  TOC,  the  Index,  and  Headings  to  locate  the 
information  they  needed. 


•  4  of  5  participants  used  the  TOC  and  subordinate  headings  extensively  to 

narrow  their  search: 

•  “I  was  looking  at  TOC  for  a  key  word....  which  chapter,  how’s  it 
going  to  break  it  down,  what  keywords  do  THEY  use... [then, 
reading  chapter  headings]. . .It’s  talking  about  principal  end 
items... I  know  I  need  to  read  through  1005-1007,  1008  too.” 

•  [After  three  failed  keyword  searches] . .  .“I’m  going  to  go  to  TOC  in 
this  case.... which  is  clearer.” 

•  [reading  TOC  headings]  “TAM  authorized  materials  that  are  where 
I’m  going  to  go.” 
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•  “....I  could  get  that  [keyword  (SL-3)]  anywhere  so  I  went  to  the 
index  to  try  and  narrow  it  down  a  lot  quicker.” 

•  “The  document  was  hard  to  negotiate... [due  to  lack  of  chapter 
headings]” 

•  5  of  5  participants  scrolled  repeatedly  between  document  locations  when 

they  had  already  determined  the  location  they  sought  from  a  TOC  or 
chapter  heading: 

•  [user  finds  TOC  and  then  pages  down  to  chapter  5,  ‘accountability 
procedures’... then  reads  chapter  headings]  “...I  might  have  been 
wrong...  Chap  ter  5  is  what  I  thought  I  wanted  but  maybe  not.” 
[scrolls  back  to  TOC] 

•  [Scrolls  from  TOC  through  to  index... TOC  led  him  to  inventory 
control  procedures.... in  index  he  found  ‘SL-3’....]  “It’s  different 
chapters.... so  I’m  going  to  go  here...”  [scrolls  back  up  to  find  the 
location] 


Users  had  to  rely  on  memory  and  workarounds  for  moving  between  references: 


•  “It  referenced  another  paragraph... I’m  going  to  go  to  that... should've 
wrote  that  down... I’m  going  to  search  that.”  [copies  paragraph  number 
and  pastes  into  search  tool] 

•  [participant  types  TOC  referenced  page  number  in  the  search  tool  to  locate 
the  page]  “...The  page  numbers  are  on  the  bottom,  but  the  forward  button 
takes  me  to  top  of  page  [so  I  can’t  see  the  page  when  I  use  it]... It’s 
delaying  the  process.” 
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Users  identified  more  efficient  ways  to  retrieve  information. 


•  “I  think  I  could  go  fas  ter....  with  a  paper  document...  the  way  I  do 

things . I  flip  to  the  front,  I  flip  to  the  back,  I  look  for  some  keywords.” 

•  I  really  like  [keyword  searching]... that  way  I  know  immediately  if  what 
I’m  looking  for  is  in  that  pub  or  I  have  to  scroll  through  it. .  .before  I  knew 
to  do  that  I  used  to  use  the  TOC  and  it  would  take  me  a  lot  longer.” 

•  “I  do  like  the  ones  where  the  TOC  is  linked.” 

•  “The  only  other  thing  is.... hyperlinking.... for  example.... I  love  when 
things  are  hyperlinked...”  [Refers  to  internal  document  reference  from 
paragraph  2011  to  paragraph  2003  (but  no  hyperlink)] 

2.  Discussion 

Primarily,  users  were  navigating  Adobe  pdf  files  that  had  produced  in  a  manner 
which  allows  keyword  searching  but  mostly  had  no  embedded  hyperlinking  (except  the 
HQMC  SOP  which  was  created  in  Microsoft  Word  format).  From  the  user  data,  it  is 
clear  that  better  navigational  aids  would  increase  efficiency.  As  one  user  indicated,  “/ 
love  when  things  are  hyperlinked.  ” 

Most  users  (4  of  5)  relied  heavily  on  the  TOC  and  subordinate  chapter  headings  to 
narrow  their  search.  However,  the  lack  of  internal  hyperlinking  or  other  navigational  aids 
resulted  in  inefficient  navigation  when  moving  between  headings  and  topic  areas.  The 
lack  of  navigational  aids  caused  users  to  scroll  a  lot,  take  notes  to  remember  document 
areas,  and  cut-and-paste  words  into  the  search  tool  to  navigate  internally.  One  user 
attempted  to  page  through  with  the  page  buttons,  but  found  that  it  was  “ delaying  the 
process.  ”  All  of  these  methods  proved  to  be  inefficient  as  users  navigated  from  topical 
headings  to  document  text  and  back  to  topical  headings.  One  user  put  it  simply,  stating, 
“/  could  go  faster  with  a  paper  document .” 
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J.  USERS  NEED  RICH  CONTEXT  TO  UNDERSTAND  SEARCH  RESULTS 

AND  IDENTIFY  KEY  INFORMATION 

Several  users  commented  that  keyword  searching,  while  beneficial,  usually  takes 
them  directly  to  a  place  in  a  document  that  does  not  allow  them  to  see  the  greater  context 
in  which  the  finding  is  presented.  This  creates  inefficiency  for  users  who  then  attempt  to 
reveal  the  context  by  scrolling  to  the  topical  heading  or  by  other  means. 

1.  User  Data 

Four  users  commented  on  the  need  for  topical  headings  to  be  associated  with 
keyword  findings: 


•  “Sometimes  the  greater  section  heading  let’s  you  know  what  context  you 
are  looking  at  that  paragraph  in.... sometimes  just  going  to  the  instance 
doesn’t  give  you  the  answer.” 

•  “If  I  went  right  to  it  [with  keyword  search]  I  might  have  to  go  back  up 
and  re-read  so  I  just  decided  to  read  the  whole  thing”  [Re-read?]  “Yeah, 
to  get  the  background...  if  you  go  right  to  that  exact  term  you  might  have 
missed  some  key  information.” 

•  [You  said,  “well,  I  need  to  go  up  a  little  bit”]  “I  wanted  to  double  check  to 
see  if  I  was  in  the  right  area.” 

•  [On  finding  a  keyword,  then  scrolling  up]  “to  see  what  are  we  talking 
about  here. . . .what  chapter  are  we  in. . .”  [scrolls  up  to  see] 

2.  Discussion 

Users  often  felt  the  need  to  “double  check  to  see  if  I  was  in  the  right  area.  ”  Users 
found  the  double-checking  inefficient  but  also  necessary  to  determine  “ what  context  you 
are  looking  at  ”  or  to  see  “ what  chapter  are  we  in.'” 
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Users  are  looking  for  contextual  infonnation  to  help  them  determine  whether  they 
have  “ missed  some  key  information,  ”  or  “to  see  what  are  we  talking  about  here.  ”  When 
the  search  tool  would  drive  users  to  the  exact  location  of  the  keyword,  with  no  contextual 
information  about  the  larger  topic  area,  users  were  then  concerned  that  it  may  be  the 
wrong  area,  or  at  least  needed  to  understand  where  they  were  within  the  document 
structure  overall. 

User  comments  also  indicate  that  seeing  information  in  context  helps  them 
accomplish  their  goals  more  effectively  and  efficiently.  Users  want  to  “get  the 
background”  and  “to  double  check”.  Research  shows  that  it  is  important  to  layer 
information  so  that  users  who  wish  to  can  access  rich  contextual  infonnation.55 
Furthermore,  context  is  beneficial  when  looking  for  the  answers  to  a  more  complex 
scenario  where  there  is  more  than  one  step  to  a  process  or  the  user  goal  is  equally 
complex. 

K.  USERS  LOOKED  FOR  INCREASED  SEARCH  FUNCTIONALITY 

Although  users  primarily  resorted  to  navigating  documents  through  the  TOC  and 
subheadings,  several  attempts  to  conduct  keyword  searches  were  unsuccessful.  The 
frustration  users  experienced  were  manifested  in  their  interview  responses,  expressions 
during  their  search,  and  the  resulting  abandonment  of  the  search  function. 


55  Beverly  B.  Zimmermann,  “Applying  Tufte's  principles  of  information  design  to  creating  effective 
Web  sites.”  In  Proceedings  of  the  15th  Annual  international  Conference  on  Computer  Documentation  (Salt 
Lake  City,  Utah,  United  States,  October  19  -  22, 1997).  SIGDOC  '97.  ACM,  New  York,  NY,  309-317. 
DOI=  http://doi.acm.org/10.1145/263367.263406 
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1. 


User  Data 


User  searches  were  ineffective: 


•  4  participants  attempted  a  total  of  17  keyword  searches  with  only  1 
useable  result  in  scenario  2 

•  Some  unsuccessful  terms  included  ‘single-purchase,’  ‘prohibited  items,’ 
and  ‘vehicle  repair’  and  in  scenario  1,  ‘CMR’ 


Users  were  frustrated  by  search  difficulties: 

•  "  ....I  can’t  type  in  ‘SL-3’  and  ‘accounting’  because  I  don’t  get 
anything. .  .In  adobe  it  will  only  find  the  search  terms  verbatim. .  .It’d  be 
neat  to  have  that  feature”  [referring  to  ability  to  search  for 
chapter/paragraph  with  multiple  search  terms] 

•  [user  reverts  to  search  box  and  types  ‘ single-purchase ’]...”I  think  it’s 
hyphenated...  [no  result]... nope... What  if  I  type  in  purchase  limit. ...ok 
it’s  not  hyphenated.” 

•  [uses  search  box  again,  typing  ‘except’... Fist  on  chin. . .hmmmmmm 
grunts]  ’’...give  me  exceptions!” 

•  “CTRL-F  wasn’t  really  working...  [uses  keyword  search  for  ‘threshold,’ 

then  ‘limit’] . none  of  those  keywords  are  in  there... I  used  a  poor 

key  word...  should  have  used  prohibited  [not  ‘unauthorized’] . that’s 

why  I  just  started  scrolling.” 
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2. 


Discussion 


Users  experienced  frustration  with  inadequate  search  results.  Specifically,  one 
limiting  factor  of  the  search  capability  is  that  it  only  returns  verbatim  matches  when  users 
expect  a  more  robust  search  that  responds  to  common  key  words  and  has  some  error 
correction.  One  user  illustrated  this,  commenting  while  searching  for  purchase-limit, 
“nope. ..What  if  I  type  in  purchase  limit. ...ok  it’s  not  hyphenated.”  Clearly,  the  user 
expects  that  the  search  tool  should  be  able  to  locate  the  term  whether  it’s  hyphenated  or 
not.  Similarly,  another  user  sought  to  find  unauthorized  items  by  using  the  keyword 
‘ unauthorized ,  ’  and  upon  no  results,  commented,  “I  used  a  poor  keyword... I  should  have 
used  ‘prohibited,  ’  that ’s  why  I just  started  scrolling .”  When  confronted  with  poor  search 
results,  users  reverted  to  other  methods  including  scrolling  and  using  a  commercially 
available  search  tool  that  did  provide  effective  search  results  (Google). 

Additionally,  users  did  not  achieve  expected  search  results  using  terms  from  their 
task.  One  user  discussed  this  limitation  and  his  desire  to  use  keywords  that  made  sense 
from  the  task  stating,  “I  can ’t  type  in  ‘SL-3  ’  and  ‘accounting  ’  because  I  don  7  get 
anything.  ”  A  robust  search  tool  that  could  provide  ‘or’  search  results,  rather  than 
searching  strictly  for  the  entire  phrase  would  have  brought  this  user  directly  to  the  ‘SL-3’ 
section  where  the  answer  was  located.  More  advanced  search  tools  even  provide  results 
based  on  relevance  of  the  relation  of  keywords  as  they  appear  in  proximity  in  the  text. 
One  user  referred  to  this,  noting,  “It ’d  be  neat  to  have  that  feature .” 

Search  tools  such  as  these  increase  the  likelihood  of  finding  the  keyword  in  the 
appropriate  context.  Additionally,  since  users  draw  keywords  from  their  tasks,  they  may 
resort  to  synonyms  or  related  terms.  A  more  advanced  search  engine  would  be  capable  of 
matching  similar  words  and  would  have  discovered  matches  for  things  like  ‘prohibited 
items.’  This  search  feature  may  greatly  assist  users  in  finding  documents. 

L.  USERS  WANT  TO  HOLD  THEIR  CURRENT  PLACE  WHILE 

CONTINUING  TO  CROSS-REFERENCE 

As  I  indicated  in  previous  findings,  users  in  most  cases  rely  on  the  TOC  and 

subheadings  to  narrow  their  search.  Additionally,  the  documents  have  internal  cross- 
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references  that  refer  to  other  information  or  locations.  In  either  case,  if  the  information  in 
the  new  location  is  not  what  users  expect,  or  if,  from  the  new  location,  they  need  to  go 
back  to  the  original  location,  they  may  not  be  able  to  find  their  way  back.  In  short,  the 
user  requires  a  method  to  get  back  to  the  point  of  origin  easily. 

1.  User  Data 

Users’  navigation  behavior  and  comments  demonstrate  inefficient  work-arounds 
to  go  back  to  previous  information: 


•  "I  feel  like  I’m  close  but. .  .It  referenced  another  paragraph. .  .I’m  going  to 
go  to  that... should've  wrote  that  down.”  [user  scrolls  between  paragraph 
2011  and  2001] 

•  Let  me  go  back  up  to  chapter  three  [to  see  what  it’s  about]... maybe  I  just 
didn’t  use  the  right  word... Let  me  go  back  to  the  top  and  look  at  my 
choices  again.” 

•  [Uses  TOC  and  then  scrolls  to  chapter  5... looking  over  chapter  headings] 
“I  might  have  been  wrong...  chap  ter  5  is  what  I  thought  I  wanted  but 
maybe  not.”  [scrolls  back  to  TOC. .  .then  chapter  2. .  .then  TOC  again] 

2.  Discussion 

The  data  reiterate  user  reliance  of  TOC  and  sub-headings  to  narrow  their  search. 
However,  users  commented,  “I  might  have  been  wrong,  ”  or  “Maybe  I  just  didn  ’t  use  the 
right  word,”  indicating  that  users  don’t  always  select  accurately  from  the  title  headings 
and  may  need  to  easily  “go  back  to  the  top  and  look  at  choices  again.”  Additionally, 
users  wanted  to  investigate  internal  cross-references  but  couldn’t  easily  do  so  because 
they  didn’t  have  a  clear  method  of  returning  to  the  current  location.  Users  instead  chose 
to  write  down  references  for  future  investigation,  since  the  capability  did  not  exist. 
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Enhancing  the  system  to  support  this  need  would  increase  the  efficiency  of  the 
information  retrieval  process.  Designers  may  consider  adding  a  “saved  file”  function  or 
supporting  internal  bookmarking,  otherwise  known  as  anchor  points,  or  anchoring. 

M.  PERSONALIZATION/CUSTOMIZATION  IS  IMPORTANT  TO  USERS 

Users  commented  that  some  information  is  referred  to  repeatedly,  and  that 
flexibility  with  paper-based  documents  allowed  them  to  tab  and  highlight  frequently  used 
information.  Users  commented  on  the  desire  to  replicate  these  features  with  electronic 
documents  by  providing  editing  tools  that  allow  highlighting  and  annotating. 

1.  User  Data 

3  of  5  users  indicated  a  desire  for  personalization  features: 


•  “I  don’t  know  if  you  can  make  notes . if  that  was  an  option  [id  like  it].” 

•  [Why  use  paper  based  over  website?]  “It’s  right  there...  It’s  easier...  I 
have  everything  tabbed. . . .1  flip  through  I  know  where  everything  is. . .” 

•  “You  use  it  so  much  we  just  had  it  printed  off. . .  .we  all  had  it  highlighted 
and  tabbed  and  marked.... for  the  majority  of  answers  you  go 
through. .  .you  go  to  the  same  stuff  over  and  over  again.” 

•  “....I  have  a  hard  copy  to  flip  through.... highlight.... for  this  particular 
document  I  would  be  all  about  that. .  .In  my  particular  publication. . .  .things 
that  are  key. . .  .1  highlight  in  the  publication.” 

2.  Discussion 

Users  stated  that  their  work  repeatedly  drives  them  to  the  same  policy  documents, 
and  the  same  locations  within  those  policy  documents.  As  one  user  stated,  “ You  go  to  the 
same  stuff  over  and  over  again."  Another  commented  on  the  frequency  of  returning  to 
common  documents,  stating,  “ you  use  it  so  much.  ”  Naturally,  users  want  to  identify 

these  common  documents  for  easy  retrieval  later. 

63 


In  the  paper-based  system,  users  were  able  to  take  advantage  of  the  physical 
presence  of  the  documents  and  could  use  tools  which  allowed  them  to  identify  the  “ things 
that  are  key’’  within  documents.  Users  typically  “ had  [the  document]  highlighted  and 
tabbed  and  marked.  ”  As  the  users  indicated,  the  ability  to  personalize  the  documents 
made  their  work  “ easier .”  Thus,  users  seek  a  capability  to  mimic  their  paper  documents 
in  order  to  replicate  this  advantage.  Adding  the  capability  to  bookmark,  highlight  and 
annotate  documents  could  prove  to  be  a  significant  enhancement  to  the  usability  of  the 
system. 
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IV.  RECOMMENDATIONS  AND  CONCLUSION 


My  analysis  concluded  that  several  actions  could  be  taken  to  improve  the  usability 
of  the  current  system.  In  this  section,  I  outline  those  recommendations  and  discuss 
possible  implementation  considerations,  limitations,  and  the  need  for  further  testing. 


A.  RECOMMENDATIONS 

Based  on  my  findings  and  applicable  industry  best  practices,  a  consolidated  listing 
of  recommendations  to  improve  the  usability  of  the  electronic  supply  policy  document 
system  is  provided  below,  followed  by  an  discussion  of  each  recommendation  in  the 
sections  that  follow. 


•  Provide  a  single  point  of  entry  to  all  documents  and  ensure  content  is 
current  and  complete 

•  Improve  the  www.usmc.mil  search  engine 

•  Provide  users  with  both  site  and  document  customization  options  that 
support  their  work 

•  Provide  a  collapsible  structure  for  navigating  and  hyperlink  internal 
references 

•  Enhance  documents  with  contextual  aids 

•  Improve  internal  document  search  capability  by  moving  away  from 
current  document  format 

•  Consider  implementing  a  more  robust  database 

•  Conduct  further  research  to  help  identify  additional  problem  areas, 
underlying  users’  needs,  and  best  practices 
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1.  Provide  a  Single  Point  of  Entry  to  All  Documents  and  Ensure  Content 
is  Current  and  Complete 

The  document  system  will  be  more  effective  if  it  allows  users  to  search  for 
documents  in  ways  that  support  their  work.  The  www.usmc.mil  website  is  an 
established,  reputable  authority  for  Marine  Corps  Directives.  However,  the  document 
collection  was  incomplete  and  documents  were  not  organized  in  ways  that  supported 
user’s  mental  models.  This  resulted  in  several  users  abandoning  the  site. 

When  going  to  the  USMC  site,  users  were  looking  for  a  complete  list  of  relevant 
documents.  My  first  recommendation  is  to  make  sure  that  the  content  on  the  site  is 
current  and  complete.  First,  include  all  types  of  documents  (a  user  manual,  Marine  Corps 
Order,  Marine  Corps  Bulletin,  P  and  non-P-type  orders,  standard  operating  procedures 
and  guidebooks,  etc.).  Second,  if  the  relevant  content  is  available  on  other  sites  and  it  is 
not  feasible  to  include  it  on  the  USMC  site,  I  recommend  that  you  provide  a  “see  also” 
feature  that  indicates  other  reliable  sources  for  each  topic.  Finally,  all  content  should  be 
dated  and  source  referenced  in  a  manner  in  which  users  can  ascertain  whether  additional 
or  more  current  information  exists  elsewhere. 

Users  demonstrated  a  variety  of  mental  models  and  key  words  for  locating 
relevant  information.  To  aid  them,  I  recommend  an  overview  page  be  designed  that 
provides  a  “one-stop  shop”  for  navigating  to  needed  content.  This  page  should  be  the 
first  option  on  the  drop-down  menu  labeled  “Publications.”  The  page  should  contain  a 
comprehensive  list  of  policy  and  procedure  documents  organized  according  to  users’ 
mental  models  rather  than  primary  office  of  responsibility  or  other  means.  Some  of  the 
users’  mental  models  uncovered  during  this  study  were  to  search  by  topic,  search  by 
SSIC  code,  and  to  view  documents  by  task  relevancy.  Further  research  is  recommended 
to  detail  these  models  more  completely. 
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2.  Improve  the  www.tismc.mil  Search  Engine 


Users  that  used  the  search  box  on  www.usmc.mil  found  it  to  be  highly  ineffective 
and  abandoned  the  site  for  better  search  capability  elsewhere.  Improving  the  search 
engine  would  make  the  site  more  effective  and  increase  users’  confidence. 

First,  users  employed  common  keywords  that  were  driven  by  the  task 
requirements.  These  common  keywords  were  logical  and  should  have  returned 
meaningful  results.  However,  the  search  results  on  the  USMC  site  did  not  do  this.  For 
example,  a  search  for  ‘ supply  manual,’’  and  another  for  ‘ accountability returned  zero 
results  even  though  the  Marine  Corps  Consumer  Level  Supply  Policy  Manual  (which 
discusses  accountability  at  length)  is  located  on  the  site.  I  recommend  that  the  search 
engine  be  improved  or  replaced  by  one  that  is  more  robust. 

Additionally,  users  expect  the  search  results  to  clearly  identify  what  parts  of  the 
site  were  searched  and  were  confused  when  it  did  not  do  this.  I  recommend  that  the 
search  filter  clearly  identify  what  area  of  the  site  was  searched.  Further,  the  distinction 
between  searching  directives  and  orders  should  be  removed  unless  it  is  significantly 
relevant  to  another  user  group. 

3.  Provide  Users  with  Both  Site  and  Document  Customization  Options 
that  Support  Their  Work 

Users  often  know  which  documents  they  need  to  consult,  and  they  want  those 
documents  to  be  available  to  them  wherever  they  go.  Providing  the  option  for  a  user  to 
create  a  policy  page  that  is  customized  to  their  preferences  would  enhance  the  user’s 
experience  and  better  support  their  work.  In  this  manner,  a  user  who  commonly  refers  to 
only  three  Marine  Corps  orders  regularly  could  have  a  personalized  policy  homepage 
where  those  three  documents  reside.  The  Federal  Acquisition  Regulation  (FAR) 
homepage  even  has  an  option  for  users  to  subscribe  to  receive  policy  update  notices  by  e- 
mail  whenever  there  is  a  change.  These  types  of  enhancements  simplify  the  users’  tasks 
when  accessing  policy  documents  and  provide  the  portability  they  desire. 
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Users  would  also  benefit  from  document  customization  options.  Specifically,  I 
recommend  that  the  site  include  the  ability  to  highlight  text  areas,  make  notes,  and 
bookmark  pages  within  a  document.  These  features  would  provide  a  useful  means  for 
users  to  replicate  their  paper-based  experience  and  build  confidence  in  using  the 
electronic  documents. 

Anchoring  and  grouping  is  another  customization  tool  that  could  provide  the 
same,  if  not  better,  functionality  as  the  ability  to  tab  pages.  Anchoring  allows  users  to 
mark  locations  in  text  for  future  retrieval  and  some  tools  allow  for  inline  annotations  as 
well.  Grouping  merely  refers  to  the  ability  of  a  user  to  ‘name’  a  group  of  anchors  that  are 
related  in  ways  that  are  meaningful  to  the  individual  user.  For  example,  if  users 
frequently  refer  to  a  document  that  discusses  UURI  accountability,  and  that  discussion  is 
spread  out  in  different  chapters  or  even  different  documents,  as  is  the  case,  a  user  could 
use  an  anchor  tool  to  mark  those  locations  and  tie  them  together  by  grouping  them  under 
a  title  ‘UURI  accounting’.  This  tool  would  allow  the  user  to  collate  information  digitally 
so  that  the  answers  to  frequently  asked  questions  could  be  referred  to  easily. 

As  a  more  long-term  goal,  consideration  may  be  given  to  a  shared  knowledge 
system  where,  collectively,  best  practices  and  interaction  within  the  user-community 
could  enhance  the  support  system  overall.  For  example,  users  in  this  study  were  asked  to 
search  for  information  on  how  to  properly  account  for  UURI  SL-3  items.  This  is  not  the 
first  time  this  debated  subject  has  plagued  a  supply  officer  and  required  him  to  refer  to 
policy  documents  to  find  the  answer.  A  shared  knowledge  system  would  not  only  give 
the  user  access  to  the  policy  documents,  but  may  alleviate  the  need  for  him  to  conduct 
exhaustive  research  for  best  practices  already  established  elsewhere. 

4.  Provide  a  Collapsible  Structure  for  Navigating  and  Hyperlinking 
Internal  References 

Users  primarily  navigated  within  documents  by  using  a  document’s  table  of 
contents  (TOC)  as  a  guide  to  get  them  to  a  section  that  they  thought  would  be  useful. 
However,  the  current  document  design  proved  to  be  inefficient  and  required  a  lot  of 
scrolling  and  returning  to  the  TOC  or  subheading  area  to  search  for  alternatives  when  the 


68 


user  did  not  find  the  information  they  needed.  Additionally,  when  references  to  another 
part  of  the  document  were  encountered  in  the  reading,  users  had  no  way  to  easily  move  to 
that  area. 

Providing  an  internal  menu  structure,  with  collapsible  headings  and  brief  content 
descriptions,  would  allow  users  to  scan  topic  headings  and  investigate  details  easily  while 
still  maintaining  an  overall  view  of  the  entire  document  structure.  Also,  providing 
hyperlinks  would  allow  users  to  investigate  other  related  material  easily.  This  is 
especially  useful  since  information  is  typically  parsed. 

5.  Enhance  Documents  with  Contextual  Aids 

Users  often  cited  that  they  needed  to  ‘check’  or  ‘double  check’  to  find  the  location 
of  the  document  in  which  they  were  currently  located.  Headings  and  subheadings  are  a 
good  way  to  indicate  this  context,  and  this  contextual  problem  can  be  partially  averted  by 
employing  a  collapsible  structure  as  discussed.  However,  contextual  cues  can  be  made  in 
other  ways  that  are  beneficial  to  the  user.  One  example  would  be  to  add  a  navigation 
frame  adjacent  to  the  document  which  shows  the  heading  and  subheading  structure  and 
highlights  the  current  location. 

Another  contextual  enhancement  option  might  be  to  provide  ‘preview’ 
information  which  would  show  the  first  few  lines  of  the  paragraph  that  a  user  was 
contemplating  when  merely  placing  the  mouse  pointer  over  the  heading.  This  would  help 
diminish  unnecessary  searching  and  would  keep  the  anticipatory  context  in  perspective. 

Whatever  contextual  aid  is  employed,  it  should  allow  the  user  to  have  an 
understanding  of  the  document  structure,  allow  them  to  see  their  current  location  within 
that  structure,  and  set  their  expectations  on  what  content  can  be  expected  when  they  go  to 
a  specific  section. 

6.  Improve  Internal  Document  Search  Capability  by  Moving  Away 
From  Current  Document  Format 

Currently,  most  electronic  policy  documents  are  scanned  versions  of  paper  policy 
documents.  Many,  but  not  all,  have  undergone  some  object  character  recognition  to 
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enable  searching.  However,  Adobe  pdf  files  have  a  finite  search  capability  that  is  limited 
by  the  Adobe  software.  Improving  the  search  capability,  then,  would  require  the  policy 
documents  to  be  migrated  to  a  source  format  that  lends  itself  to  being  searched  more 
effectively  (for  example,  html  or  xml).  Additionally,  search  engine  companies  like 
Google  have  products  available  to  enhance  document  search  capabilities.  Any  enhanced 
search  capability  considered  should  take  into  account  the  users  mental  models  and  need 
for  context  awareness  as  indicated  above. 

7.  Consider  Implementing  a  More  Robust  Database 

Users,  when  looking  for  information,  used  the  USMC  site,  Google,  and  other 
locations  that  seemed  likely  to  have  the  answers.  Not  only  do  users  want  access  to  all  the 
information  necessary  to  answer  their  questions,  the  Marine  Corps  also  wants  their 
supply  officers  to  have  infonnation  that  is  accurate.  A  consolidated  DoD  database  of 
policies  and  procedures  may  be  the  answer  to  both  of  these  needs.  This  change  may 
seem  difficult  at  the  surface  due  to  the  fact  that  many  different  agencies  are  responsible 
for  different  portions  of  the  overall  publication  system.  However,  many  industries  have 
incorporated  database  fonnats  that  allow  multiple  sources  to  be  effectively  integrated  into 
a  single  user  interface  for  presentation  to  the  end  user.  This  is  not  uncommon  and  could 
be  accomplished  for  policy  documents  as  well.  Additionally,  this  redesign  effort  should 
address  users’  goals,  tasks,  and  mental  models  consistent  with  the  findings  of  the  current 
study. 


8.  Conduct  Further  Research  to  Help  Identify  Additional  Problem 
Areas,  Underlying  Users’  Needs,  and  Best  Practices 

This  study  was  an  exploratory  study  that  revealed  a  number  of  usability  issues. 
Additional  user  testing  can  expand  our  understanding  of  the  issues  and  help  prioritize 
their  impact.  As  changes  are  made  to  the  system  within  an  overall  redesign  effort,  I 
recommend  that  on-going  testing  be  conducted.  If  a  major  redesign  is  planned,  I 
recommend  that  on-going  testing  be  combined  with  preliminary  user  needs  analysis  and 
iterative  user  input  to  help  guide  the  design  process. 
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In  addition,  I  recommend  that  the  Marine  Corps  investigate  best  practices  of 
others  to  help  guide  site  design.  One  resource  for  best  practices  is  found  on  the  website 
http://www.usability.gov,  which  provides  comprehensive  guidelines  on  the  usability  of 
web  sites.  In  addition,  I  recommend  looking  at  other  web  sites  that  are  intensively  policy 
driven.  For  instance,  some  of  the  issues  addressed  in  this  report  can  be  seen  in  a  current 
USMC  site:  USMC  Contract  Management  Process  Guide  (CMPG)  located  at 
http://hqinetOO  1  ,hq  me.  usmc.mil/i&l/v2/CMPG/i  ndcx.htm.  This  web  site  supports 
management  of  the  contracting  process,  including  the  GCPC,  and  implements  some 
features  that  may  be  useful.  Specifically,  the  CMPG  site  allows  users  to  access  policy 
guidance  in  html  format  augmented  by  links  to  other  contributing  source  information.  A 
comprehensive  flowchart  of  the  process  can  be  accessed  to  guide  users  to  the  correct  area 
where  processes  are  discussed  step-by-step  with  appropriate  references  available.  The 
context  tools  and  improved  search  capability  are  also  present  on  this  site,  which  does  not 
rely  on  Adobe  files.  Additionally,  the  site  provides  common  templates  and  other 
resources  useful  to  the  user.  The  site  is  not  customizable,  but  definitely  moves  toward 
the  knowledge-sharing  type  system  discussed  in  the  current  study. 

B.  CONCLUSION 

This  was  an  exploratory  study,  focused  on  high  level  usability  aspects  of  a 
complex  system.  In  this  study,  I  chose  a  specific  user  group  to  perfonn  specific  tasks 
using  a  ‘discount’  approach  to  usability.  I  have  shown  how  a  limited  number  of 
participants  can  reveal  many  significant  usability  problems  that  can  lead  to  improved 
design  of  a  product  or  system.  This  does  not  mean  that  the  system  is  currently  unusable 
or  that  implementation  of  these  recommendations  would  correct  all  usability  concerns. 
The  results  of  the  testing  should  be  considered  within  the  purpose  of  usability  testing 
itself-  to  make  a  product  or  system  more  usable.  To  that  end,  the  study  has  made  several 
recommendations  for  the  system  under  evaluation. 

Recommendations  with  a  low  cost-high  benefit  ratio  may  be  considered  relatively 
risk-free  to  implement  and  may  not  require  any  further  research  prior  to  implementation. 
Significant  or  costly  recommendations  may  benefit  from  further  testing  and  problem 
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triangulation,  where  data  collection  is  conducted  under  multiple  methods,  measures,  or 
approaches  to  detennine  possible  convergence  in  perceived  problem  areas.56 
Triangulation  can  be  accomplished  through  various  means  including  employing  different 
research  methods  (questionnaires,  focus  groups),  multiple  evaluators,  or  multiple  user 
groups  within  a  single  method.57 

Iterative  testing  is  recommended  either  as  a  method  of  triangulation,  to  conduct  a 
more  thorough  examination  of  a  particular  usability  area  of  interest  identified  within  the 
scope  of  this  project,  or  as  a  next  step  after  addressing  the  major  findings  of  this 
exploratory  research. 


56  Chauncey  E.  Wilson,  “Triangulation:  the  explicit  use  of  multiple  methods,  measures,  and 
approaches  for  determining  core  issues  in  product  development,”  interactions  13,  6  (November  2006),  46- 
ff.  DOI=  http://doi.acm.org/10.1145/1167948.1167980 

57  Ibid. 
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APPENDIX  A.  SUS  TEST 


Post  Test  Questionnaire 
SUS 

Mean  Score  Response  indicated  in  Bold 


Strongly  Strongly 

Disagree  Agree 


1.  I  think  that  I  would  like  to  use  this  system 
frequently 

4.2 

1 

2 

3 

4 

5 

2.  I  found  the  system  unnecessarily  complex 

1.6 

1 

2 

3 

4 

5 

3.  I  thought  the  system  was  easy  to  use 

3.2 

1 

2 

3 

4 

5 

4.  I  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  this  system 

1.2 

1 

2 

3 

4 

5 

5.  I  found  the  various  functions  in  this  system 
were  well  integrated 

3.2 

1 

2 

3 

4 

5 

6.  I  thought  there  was  too  much  inconsistency 
in  this  system 

3 

1 

2 

3 

4 

5 

7.  I  would  imagine  that  most  people  would 
learn  to  use  this  system  very  quickly 

4 

1 

2 

3 

4 

5 

8.  I  found  the  system  very  cumbersome  to  use 

1.8 

1 

2 

3 

4 

5 

9.  I  felt  very  confident  using  the  system 

3.4 

1 

2 

3 

4 

5 

10.  I  needed  to  learn  a  lot  of  things  before  I 
could  get  going  with  this  system 

1.8 
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APPENDIX  B.  SURVEY 


SUPPLY  AREA  IMPORTANCE 


1.  Do  you  consent  to  this  survey? 

Response 

Response 

Percent 

Count 

1  consent  [ 

m  99.0% 

98 

1  DO  NOT  consent  0 

1.0* 

1 

answered  question 

99 

skipped  question 

10 

2.  In  the  conduct  of  duties  as  a  supply  officer,  how  much  of  your  time  do  you  spend  looking  for  information  in  electronic 
publications? 


Response 

Response 

Percent 

Count 

<10%  1 

22  7% 

22 

10%-20%  1  1 

41.2% 

40 

20%-30%  1=  1 

20  6% 

20 

30%-50%  |  | 

124% 

12 

>50%  □ 

3.1% 

3 

answered  question 

97 

skipped  question 

12 
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3.  What  are  the  three  most  common  reasons  you  look  for  information  in  electronic  reference  publications?  (Select  only  three) 

Response 

Response 

Percent 

Count 

Develop  training  for  Subordinates  [ 

128% 

13 

Develop  internal  procedures  [ 

235% 

24 

complies  with  reference(s) 

\  539% 

55 

Advise  Commander  1 _ 

Z3  59.8% 

61 

Advise  peer(s)  /  commodity  areas  [ 

245% 

25 

. . ~ - - 

21.6% 

22 

acuon(s) 

57  8% 

59 

interested  party 

Understand  a  required  task  [ 

22  6% 

23 

Plannina  1 

17  7% 

18 

Other  (please  specify) 

0.0% 

0 

answered  question 

102 

skipped  question 

7 

4.  When  looking  for  information  in  electronic  publications,  in  which  subject  area  do  you  feel  you  have  spent  the  most  time? 

Response 

Response 

Percent 

Count 

Property  Control 

1  1 

294% 

30 

Requisition  Management 

□ 

108% 

11 

Financial  Management  (GCPC, 

1 - 1 

contracts,  budget,  fiscal  standing) 

Miscellaneous  (POR,  ammo  acctng. 

personal  effects,  publication  mgmt, 

1  1 

22  6% 

23 

clothing,  etc) 

Combat  Service  Support  (plan 

n 

4  9% 

5 

operations  support.  MPF  etc) 

l_J 

Storage  and  Distribution 

0 

2.0% 

2 

answered  question 

102 

skipped  question 

7 
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5.  When  looking  for  Property  Control  information,  which  two  tasks  require  you  to  refer  to  publications  most  often?  (select  only 
two) 


Response 

Response 

Percent 

Count 

Allowances  1 

17.2% 

5 

T/O&E  Chanqes  1 

17  2% 

5 

Redistributions  d 

3.5% 

1 

Equip  Disposal  □ 

3.5% 

1 

Controlled  Items  [ 

10  3% 

3 

Rollbacks 

0.0% 

0 

CMRs  | 

44.8% 

13 

Inventory  [ 

10.3% 

3 

Money  Value  Gain/Loss  [ 

138% 

4 

Subsidiary  records  (non-issued 

0.0% 

0 

serialized  gear) 

Crane  Reoort  1 

10.3% 

3 

Temporary  loan  files 

00% 

0 

non-T/E  assets  1  1 

6.9% 

2 

Control  of  Individual  Equipment  | _ 

138% 

4 

Investigations  [ 

27  6% 

8 

Assist  in  maintaining  MCGERR  r— ■ 

3,5% 

1 

items  ^ 

Analyze  property  accounting  reports  [ 

17.2% 

5 

Other  (please  specify)  1  1 

6.9% 

2 

answered  question 

29 

skipped  question 

80 
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6.  When  looking  for  Requisitioning  information,  which  two  tasks  require  you  to  refer  to  publications  most  often?  (select  only 

two) 

Response 

Response 

Percent 

Count 

Processing  additional  demands  , - , 

(requisitions) 

27  3% 

3 

Uniform  Movement  Issue  and 

Priority  System  (UMMIPS) 

0.0% 

0 

Maintenance  of  Requisitions  -  , - , 

45.5% 

5 

follow  up 

Manage  Daily  Maintenance  r - 1 

182% 

2 

Transactions  List  - 

Reconcile  MOV/backorder  validation  1 

18  2% 

2 

Manage  supply  discrepancy  report  [ 

27  3% 

3 

Analyze  requisition  management  , - . 

364% 

4 

reports  - 

Other  (please  specify)  [ 

9.1% 

1 

answered  question 

11 

skipped  question 

98 
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7.  When  looking  for  Financial  Management  information,  which  two  tasks  require  you  to  refer  to  publications  most  often?  (select 
only  two) 


Response 

Response 

Percent 

Count 

Prepare  budget 

1  1 

16  1% 

5 

Execute  budget/spending  plan 

1  1 

22  6% 

7 

midyear  review 

□ 

3.2% 

1 

Manage  fiscal  accountability 

□ 

9.7% 

3 

Annual  closeout 

□ 

6.5% 

2 

Manage  fiscal  for  unit  funded 

□ 

32% 

1 

exercises 

Manage  fiscal  for  externally  funded 

□ 

3.2% 

1 

exercises 

Manage  reimburseable  funding 

□ 

3.2% 

1 

Government  credit  card  program 

1 

HI  64.5% 

20 

(GCPC) 

Direct  purchases  using  the 

1 - 1 

government  credit  card 

1 _ 1 

Initiate  purchasing  and  contracting 

1 - 1 

41  9% 

13 

requests 

Manage  procurement  of  supplies 

from  DSSC 

Validate  collection  and  turn  in  of 

government  funds 

Other  (please  specify) 

□ 

3.2% 

1 

answered  question 

31 

skipped  question 
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8.  When  looking  for  Miscellaneous  information,  which  two  tasks  require  you  to  refer  to  publications  most  often?  (select  only 

two) 

Response 

Response 

Percent 

Count 

Supply  officer  and  CO  certificate  of 

1 - 1 

21  7% 

5 

relief 

Endorse  outgoing  supply  officer 

1 - 1 

8  7% 

2 

certificate  of  relief 

Letters  of  appointment  and 

1  1 

87% 

2 

revocation 

Maintain  results  of  audits  and 

l—  1 

verification 

Ij.Utv 

MLSR  reports 

1  1 

304% 

7 

Ammunition  accountability 

B 

4.4% 

1 

PORs 

□ 

8.7% 

2 

Personal  effects 

1 

!□  65.2% 

15 

Manage  permanent  issue  of  special 

0  0% 

o 

equipment 

Individual  clothing  records 

1  1 

21  7% 

5 

Publications  management 

0.0% 

0 

MOS  sustainment  training 

□ 

44% 

1 

Brief  CO  on  status  of  supply 

i - 1 

174% 

4 

operation 

Other  (please  specify) 

0.0% 

0 

answered  question 

23 

skipped  question 
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9.  When  looking  for  CSS  information,  which  two  tasks  require  you  to  refer  to  publications  most  often?  (select  only  two) 

Response 

Response 

Percent 

Count 

Develop  a  supply  support  plan  to  | - 

3 

support  MAGTF  operations  - 

Provide  supply  focused  combat 

service  support  input  in  MPF  [ 

20.0% 

1 

operations  planning 

Conduct  supply  support 

60  0% 

3 

support  element 

Preparation  of  supplies  for 

00% 

o 

deployment 

Accountability  of  unit  MPF  assets  [ 

20  0% 

i 

Establish  a  field  warehouse 

0.0% 

0 

Other  (please  specify)  E 

20  0% 

1 

answered  question 

5 

skipped  question 

104 

10.  When  looking  for  Storage  and  Distribution  information,  which  two  tasks  require  you  to  refer  to  publications  most  often? 
(select  only  two) 

Response  Response 


Maintenance  of  stored  assets  [ 

Percent 

Zl  50  0% 

50  0% 

Count 

1 

1 

100  0% 

2 

Electrostatic  discharge  sensitive  . - 

devices  - 

|  50  0% 

i 

Radiological  control  program 

00% 

0 

Other  (please  specify) 

0.0% 

0 

answered  question 

2 

skipped  question 

107 
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1 1 .  Thank  you  for  your  contribution.  Camp  Pendleton  /  San  Diego  area  personnel  only  -  we  would  appreciate  your  participation 
in  a  usability  study  (which  takes  no  more  than  one  hour).  This  study  is  being  conducted  to  better  understand  issues  in 
accessing  electronic  documents  and  supply  policies,  and  user  participation  is  critical  to  gaining  that  understanding.  If  you 
would  like  to  participate,  please  enter  your  e-mail  address  in  the  box  below  so  we  may  provide  you  with  additional  information. 
Volunteers  selected  will  be  asked  to  complete  a  task  requiring  publications  use,  followed  by  a  short  interview. 

Response 

Count 

0 

answered  question  0 

skipped  question  1 09 
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APPENDIX  C. 


PRE-TEST  QUESTIONNAIRE 


What  is  your  age  group? 

18-25 

26-35 

36-45 

46+ 

What  is  your  current  billet? 


Describe  your  familiarity  with  the  internet? 

Very  Familiar  - 1  use  it  to  look  for  information,  make  purchases,  download  applications 
Familiar  - 1  use  it  to  look  for  information,  make  purchases 
Somewhat  Familiar  - 1  use  it  primarily  to  look  for  information 

In  the  conduct  of  your  duties  as  a  supply  officer,  how  much  of  your  time  do  you  spend 
looking  for  information  in  electronic  publications? 

0%  -  10% 

10% -20% 

20%  -  30% 

30%  -  50% 

More  than  50% 

How  long  have  you  been  a  supply  officer? 

0-4  years 
4-10  years 
10  -  15  years 
15  +  years 

What  is  your  level  of  education? 

High  school  only 
Some  College 
BA/S  degree 

Some  post-graduate  education 
Graduate  degree  or  higher 
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APPENDIX  D.  TEST  PLAN 


Scope 


The  test  will  focus  on  Marine  Corps  supply  Captains  using  electronic  policy  and 
regulatory  documents  related  to  financial  management  and  property  control  functions. 

Purpose 

The  purpose  of  this  test  is  to  answer  the  research  questions: 

o  Can  users  easily  and  efficiently  locate  financial  management  and 
property  control  regulations? 

o  Can  users  easily  and  efficiently  locate  all  necessary  infonnation  to 
answer  work-related  questions? 

o  Can  users  determine  if  they  have  the  most  current  information? 
o  How  do  users  look  for  information? 


Schedule 


The  test  will  be  conducted  on  Monday,  5  November  and  Tuesday,  6  November  2007,  in 
accordance  with  the  following  schedule: 

Participant  1:  _ 

Participant  2:  _ 

Participant  3:  _ 

Participant  4:  _ 

Participant  5:  _ 

Participant  6:  _ 
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Participants 

Participants  will  represent  the  user  population,  validated  by  the  pre-test  questionnaire. 

o  Education 

Predominantly  hold  a  BA  degree,  some  higher,  but  none  lower. 

o  Job  Experience 

Between  two  and  ten  years  of  active  duty  supply  officer  experience 

o  Computer  Experience 

Familiarity  with  internet  use 

o  Demographic 

Predominantly  between  26-35  yrs  old 

Scenarios 

o  Scenario  1 

Lt  Umptyfratz,  the  coinmO  went  to  the  CO  about  a  disagreement 
with  you  on  his  CMR,  which  contains  the  following  three  items: 

TAMCN  H2067,  serial  #  4536 
TAMCN  H2067,  serial  #  4537 
TAMCN  A0267,  serial  #  M80300001 

The  Lt  claims  that  the  H2067  items  are  listed  as  SL-3  items  in  the 
TM  for  TAMCN  A0267.  Therefore,  he  says,  they  should  not  be 
listed  separately  on  his  CMR  because  he’s  already  accounting  for 
them  in  the  SL-3  inventory.  He  argues  that  it  is  the  same  situation 
as  a  MRC-138,  where  the  vehicle  is  SL-3  to  the  radio  system  and 
the  vehicle  is  not  accounted  on  the  CMR  separately. 

You  discover  through  initial  research  that  the  items  are,  in  fact, 
SL-3.  They  are  listed  as  UURI,  and  the  quantity  says  AR. 

The  CO  called  down  to  your  office  just  now:  “Captain,  I’m  headed 
to  lunch  in  five  minutes,  and  I’d  like  to  look  at  the  regulations 
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regarding  this  matter  while  I’m  out.  We’ll  discuss  it  further  when  I 
get  back.” 


•  Task  1 

Electronically  locate  all  applicable  policy/regulation 
document(s)  that  will  help  properly  guide  you  in 
accounting  for  the  SL-3  items 

•  Task  2 

“Hey  Captain,  my  lunch  partner  canceled  on  me.  Meet  me 
up  here  in  ten  minutes  to  discuss  the  CMR  situation.” 

Using  the  electronic  policy  document(s),  identify  how  to 
properly  account  for  this  SL-3  item. 

o  Scenario  2 

You  are  the  supply  officer  at  an  infantry  battalion  located  at  Camp 
Pendleton.  Your  battalion  has  been  mobilized  and  is  conducting 
work-up  training  in  29  Palms,  CA,  so  no  maintenance  personnel 
are  available.  A  GSA  vehicle  is  broken  and  requires  service.  The 
CO  doesn’t  want  to  wait  for  you  to  send  it  somewhere  on  base  for 
repair,  so  he  suggests  taking  it  to  the  commercial  GM  dealer  down 
the  road  and  using  the  GCPC  card  to  pay  for  the  service  instead. 
The  GM  dealer  estimates  the  cost  at  $2600,  but  couldn’t  guarantee 
it  wouldn’t  be  more  than  that.  The  CO  is  pretty  strict  and  always 
wants  to  see  the  exact  verbiage  and  the  reference  source  when  you 
provide  information  to  him.  Using  only  electronic  access  to  policy 
documents  as  your  source,  what  would  you  advise  the  CO? 

Key  data: 

Estimated  Cost:  $2600,  no  ceiling  price  established 
Purchase  Type:  unpriced  Service 
Item:  GSA  vehicle  repair 

•  Task  3 


The  CO  just  called  and  said  he’ll  be  down  in  five  minutes. 
He  asked  you  to  electronically  locate  all  applicable 
policy/regulation  document(s)  that  you  think  will  apply  to 
this  situation. 
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•  Task  4 


The  CO  tells  you  he  spoke  to  a  friend  who  told  him  that  the 
key  concerns  with  this  purchase  would  be  to: 

Determine  whether  the  purchase  exceeds  the  GCPC 
purchase  amount  threshold? 

Determine  if  the  GCPC  regulations/policy  prohibit  or  limit 
a  purchase  of  this  type? 

Determine  if  any  authority  prohibits  or  limits  the  purchase 
for  this  particular  item? 

•  Completion 

This  completes  scenario  2. 


Data  collected 


o  Questionnaires 

•  PRE-TEST  QUESTIONNAIRE  [APPENDIX  Cl 

The  purpose  of  the  questionnaire  is  to  record  information 
about  the  participants  of  this  study  as  they  compare  to 
characteristics  of  the  entire  user  population  to  ensure  a 
representative  group  is  tested. 

•  Post  Test  questionnaire  (SUS)  [APPENDIX  A] 

The  purpose  of  the  SUS  is  to  solicit  the  user’s  general 
assessment  of  overall  usability. 

•  Videotape 

Video  tape  all  sessions 

•  Observation  Notes 


Make  observation  notes  during  testing  to  refer  to  during 
interview 
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Interview  Questions 


Not  ‘was  it  easy’  -  can  you  talk  about  your  experience 
with....? 

Tell  me  how  you  know . how  you . 

Can  you  talk  about  that  more? 

How  confident  are  you. . . .? 

Execution 


o  Pre-test 

Setup 
o  Test 


•  Introduction 


Welcome,  and  thank  you  for  participating  today.  My  name 
is  Captain  Scott  Stahl.  I  am  a  student  at  the  Naval 
Postgraduate  School,  and  I’m  working  on  a  project  to 
evaluate  how  well  electronic  publications  assist  you  in 
doing  your  work.  As  part  of  this  project,  I  am  observing  a 
variety  of  supply  officers  conduct  various  tasks  using 
electronic  publications  to  see  what  elements  of  the  design 
might  need  to  be  improved. 

I’d  like  to  stress  that  I’m  testing  the  publication  system,  and 
not  your  abilities.  Also,  I  do  not  represent  your  command, 
or  any  other  command,  for  that  matter.  So  don’t  worry 
about  making  mistakes.  There  is  no  right  or  wrong  answer. 
If  you  find  parts  of  the  system  difficult  to  use  or 
understand,  it  is  likely  that  other  people  do  also,  and  my 
purpose  here  is  to  identify  these  items  and  make 
appropriate  recommendations  for  changes  to  improve  the 
system. 

If  you  ever  feel  that  you  are  lost  or  cannot  complete  a 
scenario  with  the  information  that  you  have  been  given, 
please  let  me  know.  I’ll  ask  you  what  you  might  do  in  a 
real-world  setting  and  then  either  put  you  on  the  right  track 
or  move  you  on  to  the  next  scenario. 
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Finally,  as  you  use  the  system,  please  do  so  as  you  would  at 
work.  I  do  ask  that  when  looking  for  information,  you  do 
so  as  quickly  and  as  accurately  as  you  can. 

Overview 


During  this  test,  I  will  first  ask  you  to  fill  out  consent  forms 
and  a  pre-test  questionnaire. 

Next,  you  will  be  presented  with  2  scenarios  containing  4 
tasks  each.  The  scenarios  will  require  you  to  navigate 
electronic  publications.  Each  scenario  will  be  presented 
individually  and  last  10  minutes.  As  you  work  through 
them,  I  will  observe  and  ask  you  to  ‘think  aloud’  as  you 
work. 

I  will  also  videotape  the  session,  recording  only  the 
computer  screen  for  analysis  in  depth  at  a  later  time. 

Upon  completion,  a  final  interview  and  questionnaire  will 
be  administered. 

In  total,  the  session  should  only  last  about  one  hour.  If  you 
would  like  to  take  a  break  at  any  time,  just  let  me  know. 

Do  you  have  any  questions  at  this  time? 

Consent  and  Pre-Test  Forms 


At  this  time,  I  would  like  to  ask  for  your  consent  to 
participate  and  be  videotaped. 

Consent  Fonn 
Videotaping  Consent 

Next,  this  pre-test  questionnaire  asks  about  characteristics 
that  will  be  compared  with  the  user  population  to  validate 
that  participants  actually  represent  the  users. 

Pre-Test  Questionnaire 

Discuss  Think  Aloud 


[think  aloud  example  here]. 


•  Initiate  Scenario  1 


At  this  time,  we  are  ready  to  begin.  You  will  be  given  10 
minutes  to  complete  the  scenario.  During  the  scenario,  if 
you  reach  a  point  where  you  would  normally  stop  and 
phone  a  colleague  for  assistance  or  quit  altogether,  please 
state  that  and  we  will  move  on  to  the  next  item. 

[ensure  videotape] 

[Present  scenario  11 

[keep  time] 

[notes] 

[end  time] 

Thank  you.  That  completes  scenario  one. 

•  Initiate  Scenario  2 


We  will  now  conduct  scenario  2.  You  will  be  given  10 
minutes  to  complete  the  scenario.  During  the  scenario,  if 
you  reach  a  point  where  you  would  normally  stop  and 
phone  a  colleague  for  assistance  or  quit  altogether,  please 
state  that  and  we  will  move  on  to  the  next  item. 

[ensure  videotape] 

[present  scenario  2] 

[keep  time] 

[notes] 

[end  time] 

Thank  you.  That  completes  scenario  two. 


Post-test 


o  Questionnaire 

I  have  a  brief  questionnaire  here  that  I’d  like  you  to  complete  also. 
The  information  you  provide  is  for  our  use  only.  Your  name  is  not 
stored  with  the  questionnaire  data. 


o  SUS  TEST 
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o  Interview 


Now  that  we  have  completed  the  scenarios,  I  have  a  few  questions 
about  what  you  experienced  during  the  test. 

Interview  Questions 


Debrief 


This  concludes  our  session.  Once  again,  I’d  like  to  say  thanks  for 
coming  today. 
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1.  Defense  Technical  Information  Center 

Fort  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Marine  Corps  Representative 
Naval  Postgraduate  School 
Monterey,  California 

4.  Director,  Training  and  Education,  MCCDC,  Code  C46 
Quantico,  Virginia 

5.  Director,  Marine  Corps  Research  Center,  MCCDC,  Code  C40RC 
Quantico,  Virginia 

6.  Marine  Corps  Tactical  Systems  Support  Activity  (Attn:  Operations 
Officer) 

Camp  Pendleton,  California 
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